]> git.mxchange.org Git - fba.git/log
fba.git
3 weeks agoContinued:
Roland Häder [Tue, 29 Apr 2025 17:19:02 +0000 (19:19 +0200)]
Continued:
- possible trolling with `eeee` as domain name and nonsense sub-domains

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 03:03:19 +0000 (05:03 +0200)]
Continued:
- added `srv.us` and `linodeusercontent.com` as mass-hosters/tunnel service,
  no real instance can be expected here
- if table `instances` doesn't bear a record then `is_recent()` should return
  `False`
- removed parameter `--single` from command `fetch_instances` and moved SQL
  statement into `else` block

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 02:42:22 +0000 (04:42 +0200)]
Continued:
- Kromonos' own Gitlab instance is the origin of this CSV file

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 02:27:26 +0000 (04:27 +0200)]
Continued:
- avoided another indenting by using `continue` for unwanted state

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 02:21:03 +0000 (04:21 +0200)]
Continued:
- fixed some tpzos (TypError -> TypeError, row[block] -> block[reason], ...)
- RuntimeException is RuntimeError (confusing as errors are hard for my understanding)
- used f-string instead (pylint is now a bit happier)
- still it is confusing with import errors?!

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 01:38:03 +0000 (03:38 +0200)]
Continued:
- renamed `_set_data()` to `_set_pending_data()`

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 01:33:07 +0000 (03:33 +0200)]
Continued:
- avoided yet another if() block while the else block contained a warning and
  used a `continue` instead for skipping non-existing headers

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 01:27:10 +0000 (03:27 +0200)]
Continued:
- reduce more `conig.get()` invocations by "caching" (locally referencing) the
  value

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 01:11:55 +0000 (03:11 +0200)]
Continued:
- also this local variable can be spared by referencing the other directly

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 01:06:54 +0000 (03:06 +0200)]
Continued:
- shorthand "e.g." replaced by "for example"
- removed if() block as a loop on an empty list is still not doing anything
  and the else block only contained a debug line

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 00:43:03 +0000 (02:43 +0200)]
Continued:
- a `SELECT` on table `instances` doesn't need to be checked if the domain is
  registered (ops)

4 weeks agoContinued:
Roland Häder [Mon, 21 Apr 2025 00:15:49 +0000 (02:15 +0200)]
Continued:
- no need to convert existing domain again, let's take row[domain] directly
- formatted SQL command

4 weeks agoContinued:
Roland Häder [Sun, 20 Apr 2025 23:55:31 +0000 (01:55 +0200)]
Continued:
- let's not shorten so much, else local functions may be confused with impored
  libraries
- renamed variable `domain` to `hostname`, not a domain only
- skip unwanted domains before invoking encode_idna()

4 weeks agoContinued:
Roland Häder [Sun, 20 Apr 2025 23:16:51 +0000 (01:16 +0200)]
Continued:
- _relays is a private array

4 weeks agoContinued:
Roland Häder [Sun, 20 Apr 2025 22:58:54 +0000 (00:58 +0200)]
Continued:
- renamed item(s) to node(s) to clarify the purpose, not generic "item"

4 weeks agoContinued:
Roland Häder [Sun, 20 Apr 2025 21:58:34 +0000 (23:58 +0200)]
Continued:
- added 'pawkey' as an alias for 'misskey'
- need to ignore unwanted domains before invoking domain_helper.encode_idna()

5 weeks agoContinued:
Roland Häder [Thu, 10 Apr 2025 15:54:32 +0000 (17:54 +0200)]
Continued:
- fedilist.com has updated their URL to https:// and no demo anymore

2 months agoContinued:
Roland Häder [Sun, 16 Mar 2025 15:47:33 +0000 (16:47 +0100)]
Continued:
- sqlite3.Row doesn't allow assigning values, so we need copy it to a dict()
  and then it can be modified

2 months agoContinued:
Roland Häder [Sat, 15 Mar 2025 00:14:24 +0000 (01:14 +0100)]
Continued:
- formatted SQL command
- added alias for misskey/pleroma

3 months agoContinued:
Roland Häder [Sat, 8 Feb 2025 15:01:48 +0000 (16:01 +0100)]
Continued:
- keyword "is" is to much here

3 months agoContinued:
Roland Häder [Thu, 6 Feb 2025 17:14:34 +0000 (18:14 +0100)]
Continued:
- delete all solved obfuscation patterns at once
- only vacuum database if rows have been deleted

3 months agoContinued:
Roland Häder [Thu, 6 Feb 2025 16:13:51 +0000 (17:13 +0100)]
Continued:
- added new command 'check_obfuscations'

3 months agoContinued:
Roland Häder [Wed, 5 Feb 2025 14:51:20 +0000 (15:51 +0100)]
Continued:
- rewrote many local variables in "row" list
- need to convert SQL row into list() row(s0
- renamed variables 'r' is to short

3 months agoContinued:
Roland Häder [Mon, 3 Feb 2025 19:12:01 +0000 (20:12 +0100)]
Continued:
- removed useless counter variable
- init local variables

3 months agoContinued:
Roland Häder [Sat, 1 Feb 2025 21:34:01 +0000 (22:34 +0100)]
Continued:
- introduced network.fetch_csv_rows()
- improved logger messages as parameter types are expected

3 months agoContinued:
Roland Häder [Sat, 1 Feb 2025 20:30:02 +0000 (21:30 +0100)]
Continued:
- command update_nodeinfo now supports --only-erroneous
- some more debug lines

3 months agoContinued:
Roland Häder [Sat, 1 Feb 2025 13:30:22 +0000 (14:30 +0100)]
Continued:
- removed duplicate

3 months agoContinued:
Roland Häder [Fri, 31 Jan 2025 21:13:09 +0000 (22:13 +0100)]
Continued:
- instances.social suddenly sends domains with TLD .arpa, need to block them

3 months agoContinued:
Roland Häder [Wed, 29 Jan 2025 15:35:58 +0000 (16:35 +0100)]
Continued:
- used wrapping network.fetch_json_rows() instead of raw network.fetch_url()

3 months agoContinued:
Roland Häder [Mon, 27 Jan 2025 01:27:34 +0000 (02:27 +0100)]
Continued:
- proper exception, not ValueException when it is a runtime exception

3 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 23:57:33 +0000 (00:57 +0100)]
Continued:
- it is a type-check, not a value-range check

3 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 22:06:13 +0000 (23:06 +0100)]
Continued:
- raise RuntimeError instead of broad Exception

3 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 14:01:46 +0000 (15:01 +0100)]
Continued:
- federation.fetch_peers() does not support relays as their "relay list"
  might be fetched differently

3 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 13:23:31 +0000 (14:23 +0100)]
Continued:
- added CSV URL from Kromonos

3 months agoContinued:
Roland Häder [Fri, 24 Jan 2025 01:14:17 +0000 (02:14 +0100)]
Continued:
- a bit more logging
- no invoking of tuple() anymore

3 months agoContinued:
Roland Häder [Wed, 22 Jan 2025 12:21:09 +0000 (13:21 +0100)]
Continued:
- all "Setting detection_mode=FOO" lines are now on info level (was mostly debug)

3 months agoContinued:
Roland Häder [Wed, 22 Jan 2025 12:06:56 +0000 (13:06 +0100)]
Continued:
- let's name columns directly where possible, instead of using their numerical
  index

3 months agoContinued:
Roland Häder [Tue, 21 Jan 2025 11:43:44 +0000 (12:43 +0100)]
Continued:
- added more exceptions to catch and handle
- split long 2-statements lines into single lines for better error handling
  and debugging (if fedilist is one day back -> https://fedilist.com )

4 months agoContinued:
Roland Häder [Sat, 18 Jan 2025 00:00:22 +0000 (01:00 +0100)]
Continued:
- default None has pleased pylint due to no dangerous default but it woud
  have made some other code (logging length of it) more complex

4 months agoContinued:
Roland Häder [Wed, 15 Jan 2025 02:36:41 +0000 (03:36 +0100)]
Continued:
- yojo-art is another alias for misskey

4 months agoContinued:
Roland Häder [Wed, 15 Jan 2025 02:12:11 +0000 (03:12 +0100)]
Continued:
- need to skip invalid table headers, they should be introduced with <thead>
  and then each column <th> but some website may use <tr> instead of <thead>
- strip (trim) strings

4 months agoContinued:
Roland Häder [Tue, 14 Jan 2025 00:48:35 +0000 (01:48 +0100)]
Continued:
- why do I have to add this unreferenced reqto?!

4 months agoContinued:
Roland Häder [Tue, 14 Jan 2025 00:17:39 +0000 (01:17 +0100)]
Continued:
- marked api_headers as private
- invoked network.fetch_url() instead of reqto.get() as the first one wraps it
  nicely

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 23:51:54 +0000 (00:51 +0100)]
Continued:
- some more PEP8 compliance by using `if 'foo' not in bar` style

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 23:40:06 +0000 (00:40 +0100)]
Continued:
- avoided dangerous (=mutable) argument to functions/methods (thanks to pylint)
- reduced invocation count of find_all("foo") by using local variable
- added more checks in "quarantined" branch

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 22:33:37 +0000 (23:33 +0100)]
Continued:
- check if required key 'url' is in dict 'row'
- avoided superflous else + indenting
- simplier check before complex checks (`if domain in domains` is lesser
  intensive than invoking "expensive" is_wanted())

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 02:57:45 +0000 (03:57 +0100)]
Continued:
- improved logging message
- also search args.software in column original_software

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 02:18:55 +0000 (03:18 +0100)]
Continued:
- reduced another return statement
- added debug line
- combined both local variable initialization

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 01:04:13 +0000 (02:04 +0100)]
Continued:
- function version_helper.remove() is now cached

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 00:41:22 +0000 (01:41 +0100)]
Continued:
- moved additional if check

4 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 00:36:41 +0000 (01:36 +0100)]
Continued:
- named named parameters

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 23:53:47 +0000 (00:53 +0100)]
Continued:
- parameter 'domain_hash' can be an empty string, so let's check it
- converted concating strings into f-string
- some debug messages added

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 22:27:20 +0000 (23:27 +0100)]
Continued:
- renamed obfuscation.is_added() to has() to streamline naming functions in
  model modules

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 21:26:53 +0000 (22:26 +0100)]
Continued:
- only invoke error_log.add() when 'write_error_log' is enabled

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 16:41:39 +0000 (17:41 +0100)]
Continued:
- fixed some pylint warnings/errors

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 14:08:41 +0000 (15:08 +0100)]
Continued:
- need to check if 'row' is type of 'dict'
- logger messages improved or added

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 03:31:10 +0000 (04:31 +0100)]
Continued:
- "cached" write_error_log configuration value

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 03:03:52 +0000 (04:03 +0100)]
Continued:
- logging whole tag isn't a good idea
- yes, the HTML is sometimes broken, e.g. <meta> and not <meta /> (self-closing)
- the open tag causes the warning

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 02:27:05 +0000 (03:27 +0100)]
Continued:
- more "caching" of config.get() calls
- yes, still good

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 02:10:42 +0000 (03:10 +0100)]
Continued:
- encoding for reading config.json set to UTF-8
- removed unused imports

4 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 00:01:13 +0000 (01:01 +0100)]
Continued:
- added missing type 'bool'
- improved/added debug lines

4 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 23:18:35 +0000 (00:18 +0100)]
Continued:
- removed old-lost code utils.fetch_url(), maybe a leftover from previous
  moving code to network module?
- rewrote loop (better error handling

4 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 22:44:50 +0000 (23:44 +0100)]
Continued:
- messages fixed

4 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 22:02:33 +0000 (23:02 +0100)]
Continued:
- copy fetchall() result into local variable for counting total rows

4 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 20:50:03 +0000 (21:50 +0100)]
Continued:
- need to convert sqlite3.Row into dict to be able to change values

4 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 20:41:45 +0000 (21:41 +0100)]
Continued:
- ops, was conflicting name
- added debug message

4 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 19:12:13 +0000 (20:12 +0100)]
Continued:
- rewrote multiple variables to array
- renamed row -> obfuscated to make space for above change

4 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 15:05:55 +0000 (16:05 +0100)]
Continued:
- lets better move these values to a central place

4 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 09:02:45 +0000 (10:02 +0100)]
Continued:
- moved timeout values for nodeinfo-fetch to local "cached" variable

4 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:41:22 +0000 (09:41 +0100)]
Continued:
- blacklisted pinggy.link, this floods instances table. Please register yourself
  a proper domain and not use "free" hosters' server names as "domain name"

4 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:38:38 +0000 (09:38 +0100)]
Continued:
- introduced network.get_generic() to encapsulate some repeated parameters

4 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:19:55 +0000 (09:19 +0100)]
Continued:
- software.remove_extras() is now cached

4 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:15:04 +0000 (09:15 +0100)]
Continued:
- added commented out debug lines

4 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 07:36:37 +0000 (08:36 +0100)]
Continued:
- moved alias_block_level() to new fba/helpers/blocks.py

4 months agoContinued:
Roland Häder [Thu, 9 Jan 2025 01:14:14 +0000 (02:14 +0100)]
Continued:
- added needed timeout parameter (pylint)
- removed unused imports (pylint)

4 months agoContinued:
Roland Häder [Tue, 7 Jan 2025 03:31:37 +0000 (04:31 +0100)]
Continued:
- local "caching" of configuration values to speedup code execution and lowering
  massive debug logging
- combined None and "" together

4 months agoContinued:
Roland Häder [Sat, 4 Jan 2025 00:49:52 +0000 (01:49 +0100)]
Continued:
- some variables fixed (e.g. no row[domain] but source_domain does exist
- added some random sleep for fetch_observer() as they seem to block to quick
  responses on automatic requests

4 months agoContinued:
Roland Häder [Fri, 3 Jan 2025 02:36:12 +0000 (03:36 +0100)]
Continued:
- fixed deprecation warning for pylint 3.0 ending support for short class names
- wrong %d for a string fixed

4 months agoContinued:
Roland Häder [Mon, 30 Dec 2024 04:32:14 +0000 (05:32 +0100)]
Continued:
- introduced software_helper.remove_extras()

4 months agoContinued:
Roland Häder [Sun, 29 Dec 2024 22:18:45 +0000 (23:18 +0100)]
Continued:
- need to createa list() of CSV reader object to be able to count total CSV
  lines

5 months agoContinued:
Roland Häder [Sat, 7 Dec 2024 05:24:02 +0000 (06:24 +0100)]
Continued:
- full-stop (dot) no longer at end of exception messages
- added column last_requested_path to 'instances' table

6 months agoContinued:
Roland Häder [Fri, 15 Nov 2024 00:15:25 +0000 (01:15 +0100)]
Continued:
- better handling of CSRF meta tags

6 months agoContinued:
Roland Häder [Fri, 8 Nov 2024 20:26:59 +0000 (21:26 +0100)]
Continued:
- fixed some leftovers from last encapsulation

6 months agoContinued:
Roland Häder [Fri, 8 Nov 2024 13:58:44 +0000 (14:58 +0100)]
Continued:
- maybe "index" was outdated here?

7 months agoContionued:
Roland Häder [Sun, 13 Oct 2024 14:01:17 +0000 (16:01 +0200)]
Contionued:
- moved static array from code (body) to header

7 months agoContinued:
Roland Häder [Fri, 11 Oct 2024 14:59:03 +0000 (16:59 +0200)]
Continued:
- fixed exception: need to skip unwanted TLDs

7 months agoContinued:
Roland Häder [Sat, 5 Oct 2024 20:04:40 +0000 (22:04 +0200)]
Continued:
- added locally used scripts, can be useful to someone else, too

7 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 20:05:59 +0000 (22:05 +0200)]
Continued:
- a little commata must be added ... :-(

7 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 20:02:29 +0000 (22:02 +0200)]
Continued:
- removed no longer provided or now private blocklist

7 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 19:47:25 +0000 (21:47 +0200)]
Continued:
- removed redundant code (copy-paste)
- added missing local variable declaration

7 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 17:00:37 +0000 (19:00 +0200)]
Continued:
- now also command recheck_obfuscation() is updating/adding 'obfucated' a record

7 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 16:52:05 +0000 (18:52 +0200)]
Continued:
- added function update() for "model" obfuscation (module name and table name
  must match as this is naming convention)
- added column 'last_used' to table 'obfuscation'

7 months agoContinued:
Roland Häder [Mon, 30 Sep 2024 19:23:42 +0000 (21:23 +0200)]
Continued:
- changed debug to info message as this was to quiet when for example the new
  instance isn't reachable (timeout)

7 months agoContinued:
Roland Häder [Sun, 29 Sep 2024 04:11:32 +0000 (06:11 +0200)]
Continued:
- added debug lines for function invocation
- change INFO into DEBUG message
- changed type to "any" (maybe it is limitable?)

7 months agoContinued:
Roland Häder [Sat, 21 Sep 2024 23:28:59 +0000 (01:28 +0200)]
Continued:
- more fixed for handing parked domains (200 OK or 302 but no real content)

7 months agoContinued:
Roland Häder [Sat, 21 Sep 2024 23:13:57 +0000 (01:13 +0200)]
Continued:
- some parked domains return "200 OK" for all requests. This is understandable
  in their view on things but it isn't standard-conform as 200 means document
  found and not some fake content being returned

8 months agoContinued:
Roland Häder [Thu, 19 Sep 2024 21:50:20 +0000 (23:50 +0200)]
Continued:
- removed obsolete code

8 months agoContinued:
Roland Häder [Thu, 19 Sep 2024 17:24:02 +0000 (19:24 +0200)]
Continued:
- an empty string is not the same as None, that's correct. So both need to be
  checked
- yes, first I check on false (missing keys, etc.) then true (positive outcome)

8 months agoContinued:
Roland Häder [Wed, 18 Sep 2024 22:37:32 +0000 (00:37 +0200)]
Continued:
- introduced network.fetch_json_rows() which fetches a JSON API response and
  returns a specifyable rows key (optional) to return as rows

8 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 17:17:41 +0000 (19:17 +0200)]
Continued:
- introduced domain_helper.is_tld_wanted() which checks if the top-level domain
  of a domain name is wanted