]> git.mxchange.org Git - fba.git/log
fba.git
12 days agoContinued: master
Roland Häder [Sun, 16 Mar 2025 15:47:33 +0000 (16:47 +0100)]
Continued:
- sqlite3.Row doesn't allow assigning values, so we need copy it to a dict()
  and then it can be modified

2 weeks agoContinued:
Roland Häder [Sat, 15 Mar 2025 00:14:24 +0000 (01:14 +0100)]
Continued:
- formatted SQL command
- added alias for misskey/pleroma

6 weeks agoContinued:
Roland Häder [Sat, 8 Feb 2025 15:01:48 +0000 (16:01 +0100)]
Continued:
- keyword "is" is to much here

7 weeks agoContinued:
Roland Häder [Thu, 6 Feb 2025 17:14:34 +0000 (18:14 +0100)]
Continued:
- delete all solved obfuscation patterns at once
- only vacuum database if rows have been deleted

7 weeks agoContinued:
Roland Häder [Thu, 6 Feb 2025 16:13:51 +0000 (17:13 +0100)]
Continued:
- added new command 'check_obfuscations'

7 weeks agoContinued:
Roland Häder [Wed, 5 Feb 2025 14:51:20 +0000 (15:51 +0100)]
Continued:
- rewrote many local variables in "row" list
- need to convert SQL row into list() row(s0
- renamed variables 'r' is to short

7 weeks agoContinued:
Roland Häder [Mon, 3 Feb 2025 19:12:01 +0000 (20:12 +0100)]
Continued:
- removed useless counter variable
- init local variables

7 weeks agoContinued:
Roland Häder [Sat, 1 Feb 2025 21:34:01 +0000 (22:34 +0100)]
Continued:
- introduced network.fetch_csv_rows()
- improved logger messages as parameter types are expected

7 weeks agoContinued:
Roland Häder [Sat, 1 Feb 2025 20:30:02 +0000 (21:30 +0100)]
Continued:
- command update_nodeinfo now supports --only-erroneous
- some more debug lines

7 weeks agoContinued:
Roland Häder [Sat, 1 Feb 2025 13:30:22 +0000 (14:30 +0100)]
Continued:
- removed duplicate

8 weeks agoContinued:
Roland Häder [Fri, 31 Jan 2025 21:13:09 +0000 (22:13 +0100)]
Continued:
- instances.social suddenly sends domains with TLD .arpa, need to block them

8 weeks agoContinued:
Roland Häder [Wed, 29 Jan 2025 15:35:58 +0000 (16:35 +0100)]
Continued:
- used wrapping network.fetch_json_rows() instead of raw network.fetch_url()

2 months agoContinued:
Roland Häder [Mon, 27 Jan 2025 01:27:34 +0000 (02:27 +0100)]
Continued:
- proper exception, not ValueException when it is a runtime exception

2 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 23:57:33 +0000 (00:57 +0100)]
Continued:
- it is a type-check, not a value-range check

2 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 22:06:13 +0000 (23:06 +0100)]
Continued:
- raise RuntimeError instead of broad Exception

2 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 14:01:46 +0000 (15:01 +0100)]
Continued:
- federation.fetch_peers() does not support relays as their "relay list"
  might be fetched differently

2 months agoContinued:
Roland Häder [Sun, 26 Jan 2025 13:23:31 +0000 (14:23 +0100)]
Continued:
- added CSV URL from Kromonos

2 months agoContinued:
Roland Häder [Fri, 24 Jan 2025 01:14:17 +0000 (02:14 +0100)]
Continued:
- a bit more logging
- no invoking of tuple() anymore

2 months agoContinued:
Roland Häder [Wed, 22 Jan 2025 12:21:09 +0000 (13:21 +0100)]
Continued:
- all "Setting detection_mode=FOO" lines are now on info level (was mostly debug)

2 months agoContinued:
Roland Häder [Wed, 22 Jan 2025 12:06:56 +0000 (13:06 +0100)]
Continued:
- let's name columns directly where possible, instead of using their numerical
  index

2 months agoContinued:
Roland Häder [Tue, 21 Jan 2025 11:43:44 +0000 (12:43 +0100)]
Continued:
- added more exceptions to catch and handle
- split long 2-statements lines into single lines for better error handling
  and debugging (if fedilist is one day back -> https://fedilist.com )

2 months agoContinued:
Roland Häder [Sat, 18 Jan 2025 00:00:22 +0000 (01:00 +0100)]
Continued:
- default None has pleased pylint due to no dangerous default but it woud
  have made some other code (logging length of it) more complex

2 months agoContinued:
Roland Häder [Wed, 15 Jan 2025 02:36:41 +0000 (03:36 +0100)]
Continued:
- yojo-art is another alias for misskey

2 months agoContinued:
Roland Häder [Wed, 15 Jan 2025 02:12:11 +0000 (03:12 +0100)]
Continued:
- need to skip invalid table headers, they should be introduced with <thead>
  and then each column <th> but some website may use <tr> instead of <thead>
- strip (trim) strings

2 months agoContinued:
Roland Häder [Tue, 14 Jan 2025 00:48:35 +0000 (01:48 +0100)]
Continued:
- why do I have to add this unreferenced reqto?!

2 months agoContinued:
Roland Häder [Tue, 14 Jan 2025 00:17:39 +0000 (01:17 +0100)]
Continued:
- marked api_headers as private
- invoked network.fetch_url() instead of reqto.get() as the first one wraps it
  nicely

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 23:51:54 +0000 (00:51 +0100)]
Continued:
- some more PEP8 compliance by using `if 'foo' not in bar` style

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 23:40:06 +0000 (00:40 +0100)]
Continued:
- avoided dangerous (=mutable) argument to functions/methods (thanks to pylint)
- reduced invocation count of find_all("foo") by using local variable
- added more checks in "quarantined" branch

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 22:33:37 +0000 (23:33 +0100)]
Continued:
- check if required key 'url' is in dict 'row'
- avoided superflous else + indenting
- simplier check before complex checks (`if domain in domains` is lesser
  intensive than invoking "expensive" is_wanted())

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 02:57:45 +0000 (03:57 +0100)]
Continued:
- improved logging message
- also search args.software in column original_software

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 02:18:55 +0000 (03:18 +0100)]
Continued:
- reduced another return statement
- added debug line
- combined both local variable initialization

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 01:04:13 +0000 (02:04 +0100)]
Continued:
- function version_helper.remove() is now cached

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 00:41:22 +0000 (01:41 +0100)]
Continued:
- moved additional if check

2 months agoContinued:
Roland Häder [Mon, 13 Jan 2025 00:36:41 +0000 (01:36 +0100)]
Continued:
- named named parameters

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 23:53:47 +0000 (00:53 +0100)]
Continued:
- parameter 'domain_hash' can be an empty string, so let's check it
- converted concating strings into f-string
- some debug messages added

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 22:27:20 +0000 (23:27 +0100)]
Continued:
- renamed obfuscation.is_added() to has() to streamline naming functions in
  model modules

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 21:26:53 +0000 (22:26 +0100)]
Continued:
- only invoke error_log.add() when 'write_error_log' is enabled

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 16:41:39 +0000 (17:41 +0100)]
Continued:
- fixed some pylint warnings/errors

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 14:08:41 +0000 (15:08 +0100)]
Continued:
- need to check if 'row' is type of 'dict'
- logger messages improved or added

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 03:31:10 +0000 (04:31 +0100)]
Continued:
- "cached" write_error_log configuration value

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 03:03:52 +0000 (04:03 +0100)]
Continued:
- logging whole tag isn't a good idea
- yes, the HTML is sometimes broken, e.g. <meta> and not <meta /> (self-closing)
- the open tag causes the warning

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 02:27:05 +0000 (03:27 +0100)]
Continued:
- more "caching" of config.get() calls
- yes, still good

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 02:10:42 +0000 (03:10 +0100)]
Continued:
- encoding for reading config.json set to UTF-8
- removed unused imports

2 months agoContinued:
Roland Häder [Sun, 12 Jan 2025 00:01:13 +0000 (01:01 +0100)]
Continued:
- added missing type 'bool'
- improved/added debug lines

2 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 23:18:35 +0000 (00:18 +0100)]
Continued:
- removed old-lost code utils.fetch_url(), maybe a leftover from previous
  moving code to network module?
- rewrote loop (better error handling

2 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 22:44:50 +0000 (23:44 +0100)]
Continued:
- messages fixed

2 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 22:02:33 +0000 (23:02 +0100)]
Continued:
- copy fetchall() result into local variable for counting total rows

2 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 20:50:03 +0000 (21:50 +0100)]
Continued:
- need to convert sqlite3.Row into dict to be able to change values

2 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 20:41:45 +0000 (21:41 +0100)]
Continued:
- ops, was conflicting name
- added debug message

2 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 19:12:13 +0000 (20:12 +0100)]
Continued:
- rewrote multiple variables to array
- renamed row -> obfuscated to make space for above change

2 months agoContinued:
Roland Häder [Sat, 11 Jan 2025 15:05:55 +0000 (16:05 +0100)]
Continued:
- lets better move these values to a central place

2 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 09:02:45 +0000 (10:02 +0100)]
Continued:
- moved timeout values for nodeinfo-fetch to local "cached" variable

2 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:41:22 +0000 (09:41 +0100)]
Continued:
- blacklisted pinggy.link, this floods instances table. Please register yourself
  a proper domain and not use "free" hosters' server names as "domain name"

2 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:38:38 +0000 (09:38 +0100)]
Continued:
- introduced network.get_generic() to encapsulate some repeated parameters

2 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:19:55 +0000 (09:19 +0100)]
Continued:
- software.remove_extras() is now cached

2 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 08:15:04 +0000 (09:15 +0100)]
Continued:
- added commented out debug lines

2 months agoContinued:
Roland Häder [Fri, 10 Jan 2025 07:36:37 +0000 (08:36 +0100)]
Continued:
- moved alias_block_level() to new fba/helpers/blocks.py

2 months agoContinued:
Roland Häder [Thu, 9 Jan 2025 01:14:14 +0000 (02:14 +0100)]
Continued:
- added needed timeout parameter (pylint)
- removed unused imports (pylint)

2 months agoContinued:
Roland Häder [Tue, 7 Jan 2025 03:31:37 +0000 (04:31 +0100)]
Continued:
- local "caching" of configuration values to speedup code execution and lowering
  massive debug logging
- combined None and "" together

2 months agoContinued:
Roland Häder [Sat, 4 Jan 2025 00:49:52 +0000 (01:49 +0100)]
Continued:
- some variables fixed (e.g. no row[domain] but source_domain does exist
- added some random sleep for fetch_observer() as they seem to block to quick
  responses on automatic requests

2 months agoContinued:
Roland Häder [Fri, 3 Jan 2025 02:36:12 +0000 (03:36 +0100)]
Continued:
- fixed deprecation warning for pylint 3.0 ending support for short class names
- wrong %d for a string fixed

2 months agoContinued:
Roland Häder [Mon, 30 Dec 2024 04:32:14 +0000 (05:32 +0100)]
Continued:
- introduced software_helper.remove_extras()

2 months agoContinued:
Roland Häder [Sun, 29 Dec 2024 22:18:45 +0000 (23:18 +0100)]
Continued:
- need to createa list() of CSV reader object to be able to count total CSV
  lines

3 months agoContinued:
Roland Häder [Sat, 7 Dec 2024 05:24:02 +0000 (06:24 +0100)]
Continued:
- full-stop (dot) no longer at end of exception messages
- added column last_requested_path to 'instances' table

4 months agoContinued:
Roland Häder [Fri, 15 Nov 2024 00:15:25 +0000 (01:15 +0100)]
Continued:
- better handling of CSRF meta tags

4 months agoContinued:
Roland Häder [Fri, 8 Nov 2024 20:26:59 +0000 (21:26 +0100)]
Continued:
- fixed some leftovers from last encapsulation

4 months agoContinued:
Roland Häder [Fri, 8 Nov 2024 13:58:44 +0000 (14:58 +0100)]
Continued:
- maybe "index" was outdated here?

5 months agoContionued:
Roland Häder [Sun, 13 Oct 2024 14:01:17 +0000 (16:01 +0200)]
Contionued:
- moved static array from code (body) to header

5 months agoContinued:
Roland Häder [Fri, 11 Oct 2024 14:59:03 +0000 (16:59 +0200)]
Continued:
- fixed exception: need to skip unwanted TLDs

5 months agoContinued:
Roland Häder [Sat, 5 Oct 2024 20:04:40 +0000 (22:04 +0200)]
Continued:
- added locally used scripts, can be useful to someone else, too

5 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 20:05:59 +0000 (22:05 +0200)]
Continued:
- a little commata must be added ... :-(

5 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 20:02:29 +0000 (22:02 +0200)]
Continued:
- removed no longer provided or now private blocklist

5 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 19:47:25 +0000 (21:47 +0200)]
Continued:
- removed redundant code (copy-paste)
- added missing local variable declaration

5 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 17:00:37 +0000 (19:00 +0200)]
Continued:
- now also command recheck_obfuscation() is updating/adding 'obfucated' a record

5 months agoContinued:
Roland Häder [Tue, 1 Oct 2024 16:52:05 +0000 (18:52 +0200)]
Continued:
- added function update() for "model" obfuscation (module name and table name
  must match as this is naming convention)
- added column 'last_used' to table 'obfuscation'

5 months agoContinued:
Roland Häder [Mon, 30 Sep 2024 19:23:42 +0000 (21:23 +0200)]
Continued:
- changed debug to info message as this was to quiet when for example the new
  instance isn't reachable (timeout)

5 months agoContinued:
Roland Häder [Sun, 29 Sep 2024 04:11:32 +0000 (06:11 +0200)]
Continued:
- added debug lines for function invocation
- change INFO into DEBUG message
- changed type to "any" (maybe it is limitable?)

6 months agoContinued:
Roland Häder [Sat, 21 Sep 2024 23:28:59 +0000 (01:28 +0200)]
Continued:
- more fixed for handing parked domains (200 OK or 302 but no real content)

6 months agoContinued:
Roland Häder [Sat, 21 Sep 2024 23:13:57 +0000 (01:13 +0200)]
Continued:
- some parked domains return "200 OK" for all requests. This is understandable
  in their view on things but it isn't standard-conform as 200 means document
  found and not some fake content being returned

6 months agoContinued:
Roland Häder [Thu, 19 Sep 2024 21:50:20 +0000 (23:50 +0200)]
Continued:
- removed obsolete code

6 months agoContinued:
Roland Häder [Thu, 19 Sep 2024 17:24:02 +0000 (19:24 +0200)]
Continued:
- an empty string is not the same as None, that's correct. So both need to be
  checked
- yes, first I check on false (missing keys, etc.) then true (positive outcome)

6 months agoContinued:
Roland Häder [Wed, 18 Sep 2024 22:37:32 +0000 (00:37 +0200)]
Continued:
- introduced network.fetch_json_rows() which fetches a JSON API response and
  returns a specifyable rows key (optional) to return as rows

6 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 17:17:41 +0000 (19:17 +0200)]
Continued:
- introduced domain_helper.is_tld_wanted() which checks if the top-level domain
  of a domain name is wanted

6 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 16:03:34 +0000 (18:03 +0200)]
Continued:
- skip invalid domains after checking them for obfuscation with asterisks or
  question marks

6 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 15:29:42 +0000 (17:29 +0200)]
Continued:
- rewrote chaos.social parser for their own documentation at meta.chaos.social

6 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 10:35:19 +0000 (12:35 +0200)]
Continued:
- instances.delete() now has an optional parameter 'column' which
  also needs validation on type and non-empty string
- some functions in same module didn't log optional/extra parameter 'column'

6 months agoContinued:
Roland Häder [Sun, 15 Sep 2024 10:23:43 +0000 (12:23 +0200)]
Continued:
- these DELETE SQL statements have already been encapsulated

6 months agoContinued:
Roland Häder [Sat, 14 Sep 2024 18:43:56 +0000 (20:43 +0200)]
Continued:
- formatted longer SQL statment
- changed ordering of records

6 months agoContinued:
Roland Häder [Wed, 11 Sep 2024 23:29:52 +0000 (01:29 +0200)]
Continued:
- 'typeecho' sometimes provide a block list which has same API path as Fediverse
  instances provide, so let's add it as a source
- removed nodeinfo_url from SQL query when searching for software during
  fetch_blocks() command

6 months agoContinued:
Roland Häder [Sun, 8 Sep 2024 03:02:52 +0000 (05:02 +0200)]
Continued:
- added "logging" of obfuscated blocked domain/instance names
- if it has been deobfuscated, it will be removed

6 months agoContinued:
Roland Häder [Sun, 8 Sep 2024 00:16:26 +0000 (02:16 +0200)]
Continued:
- check if instance[host] is at least set and a non-empty string

6 months agoContinued:
Roland Häder [Sat, 7 Sep 2024 17:14:20 +0000 (19:14 +0200)]
Continued:
- when returned value from tidyup.domain() is an empty string or None then
  always log it as a debug message

6 months agoContinued:
Roland Häder [Wed, 4 Sep 2024 00:20:34 +0000 (02:20 +0200)]
Continued:
- added type-hint 'None' for all functions not returning a value, for example
  setters, add/update/delete functions

6 months agoContinued:
Roland Häder [Tue, 3 Sep 2024 23:07:37 +0000 (01:07 +0200)]
Continued:
- don't log an info after a "filter function"
- initialize variable 'search' as both if() blocks use the same code
- check variable 'domain' before 'char' as 'char' needs to be in a supported
  'domain' string

6 months agoContinued:
Roland Häder [Sat, 31 Aug 2024 18:12:20 +0000 (20:12 +0200)]
Continued:
- added another alias 'meowkey' for 'misskey'

7 months agoContinued:
Roland Häder [Sat, 24 Aug 2024 17:53:18 +0000 (19:53 +0200)]
Continued:
- Diaspora seem to provide an API for more peers, so let's add it for command
  fetch_instances()
- ... and yes, that long query isn't a good idea

7 months agoContinued:
Roland Häder [Fri, 23 Aug 2024 21:16:26 +0000 (23:16 +0200)]
Continued:
- added alias 'barkle' for 'misskey'

7 months agoContinued:
Roland Häder [Thu, 15 Aug 2024 19:35:57 +0000 (21:35 +0200)]
Continued:
- let also process.block() abort when an unwanted block_level is provided
- alias also "reject" to "rejected"

7 months agoContinued:
Roland Häder [Tue, 13 Aug 2024 21:22:01 +0000 (23:22 +0200)]
Continued:
- script is rejecting "reject" now in favor of "rejected" as there is
  "suspended" and "silenced" (past) already around
- Please run `$ sqlite3 blocks.db "UPDATE blocks SET block_level = 'rejected' WHERE block_level = 'reject';"`

7 months agoContinued:
Roland Häder [Wed, 7 Aug 2024 15:24:23 +0000 (17:24 +0200)]
Continued:
- update_nodeinfo() now accepts --force-recrawl and --force-update-none to have
  both "special" modes separately usable