]> git.mxchange.org Git - fba.git/log
fba.git
14 months agoContinued:
Roland Häder [Thu, 31 Aug 2023 06:00:52 +0000 (08:00 +0200)]
Continued:
- notice added about difference between total peer/block count

14 months agoContinued:
Roland Häder [Thu, 31 Aug 2023 05:53:16 +0000 (07:53 +0200)]
Continued:
- during fetching block lists or peers, filter out already blacklisted
  entries which will keep the list variable's memory usage smaller

14 months agoContinued:
Roland Häder [Wed, 30 Aug 2023 06:32:24 +0000 (08:32 +0200)]
Continued:
- formatted template block
- added average blocks

14 months agoContinued:
Roland Häder [Wed, 30 Aug 2023 01:31:34 +0000 (03:31 +0200)]
Continued:
- need to check blacklist/is_wanted()-check first before invoking
  blocks.is_instance_blocked()

14 months agoContinued:
Roland Häder [Tue, 29 Aug 2023 23:10:49 +0000 (01:10 +0200)]
Continued:
- removed a lot redundant messages, can be easier

14 months agoContinued:
Roland Häder [Tue, 29 Aug 2023 09:23:39 +0000 (11:23 +0200)]
Continued:
- added checks if instance is registered when update function are invoked
- some empty lines added

14 months agoContinued:
Roland Häder [Tue, 29 Aug 2023 06:46:05 +0000 (08:46 +0200)]
Continued:
- removed no longer used import

14 months agoAnother attempt to rewrite:
Roland Häder [Tue, 29 Aug 2023 05:52:42 +0000 (07:52 +0200)]
Another attempt to rewrite:
- don't update nodeinfo URL and detection mode to STATIC_CHECK while fetching
  blocks for Pleroma
- Pleroma has their block list exposed in that nodeinfo and not in separate API

14 months agoContinued:
Roland Häder [Mon, 28 Aug 2023 14:30:09 +0000 (16:30 +0200)]
Continued:
- some misskey instances may have no nodeinfo URL, e.g. they got only detected
  through APP_NAME method
- still they may provide a blocklist
- it is now rewritten that first a generic "/api/v1/instance/domain_blocks" is
  fetched and if it fails, a software-specific attempt is done

14 months agoContinued:
Roland Häder [Mon, 28 Aug 2023 12:11:50 +0000 (14:11 +0200)]
Continued:
- also need to break the loop when no 'data' element is being returned which
  might be the case when the server doesn't return a JSON

14 months agoContinued:
Roland Häder [Sun, 27 Aug 2023 14:27:54 +0000 (16:27 +0200)]
Continued:
- let's try it stricter: 200 OK is the only HTTP status code we accept
- log failing fetch of /instances from a Lemmy instance

14 months agoOps:
Roland Häder [Sun, 27 Aug 2023 06:02:01 +0000 (08:02 +0200)]
Ops:
- need to be negated
- moved LICENSE file to docs

14 months agoContinued:
Roland Häder [Sat, 26 Aug 2023 21:47:29 +0000 (23:47 +0200)]
Continued:
- if/elif/raise blocks should be separated from other logic
- tpzo fixed

14 months agoContinued:
Roland Häder [Sat, 26 Aug 2023 21:03:37 +0000 (23:03 +0200)]
Continued:
- warn about invalid SHA256 domain digests (aka. hashes)
- some mastodon instances maybe not include them at all?

14 months agoContinued:
Roland Häder [Sat, 26 Aug 2023 20:57:45 +0000 (22:57 +0200)]
Continued:
- fixed some pylint issues

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 23:31:52 +0000 (01:31 +0200)]
Continued:
- I hope this isn't to strict, some hosts return a "298 None" which the HTTP
  library doesn't see as failed (response.ok = False) but still doesn't return
  a JSON

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 19:41:11 +0000 (21:41 +0200)]
Continued:
- exclude NULL values for detection_mode

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 18:37:56 +0000 (20:37 +0200)]
Continued:
- half day for rechecking block lists was to fast
- also shortened API last access

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 15:53:11 +0000 (17:53 +0200)]
Continued:
- reformatted a bit

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 15:50:18 +0000 (17:50 +0200)]
Continued:
- fixed Shell script
- scoreboard for commands is total peers

14 months agoContinued:
Roland Häder [Fri, 25 Aug 2023 01:02:21 +0000 (03:02 +0200)]
Continued:
- renamed update_data() to update()

14 months agoSorting order changed:
Roland Häder [Thu, 24 Aug 2023 22:07:51 +0000 (00:07 +0200)]
Sorting order changed:
- first total_peers/blocks descending
- then last_updated ascending

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 20:46:18 +0000 (22:46 +0200)]
Continued:
- debug log message added

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 20:35:33 +0000 (22:35 +0200)]
Continued:
- added alias 'revolver' as this is another one for pleroma

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 20:29:19 +0000 (22:29 +0200)]
Continued:
- fetch all data in /list view
- added total_peers and total_blocks to be exposed

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 19:12:14 +0000 (21:12 +0200)]
Continued:
- added parameter --software3 which searchinf for a file 'software.txt'
- you can generate this by running e.g.
  sqlite3 ./blocks.db "SELECT software FROM instances WHAT_EVER_PARAMETER;" > software.txt
- reset nodeinfo_url if software is now None
- always set complete URL, including domain

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 18:35:55 +0000 (20:35 +0200)]
Continued:
- renamed fetch_nodeinfo() to fetch() as it is already part of module nodeinfo
- added 3rd optional parameter to it, fetching of /.well-known/* isn't then
  required anymore and saves another request
- also the wanted URL can be directly used

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 16:59:38 +0000 (18:59 +0200)]
Continued:
- also order this list by total peers first, then row id

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 09:42:54 +0000 (11:42 +0200)]
Continued:
- also order by rowid descending

14 months agoContinued:
Roland Häder [Thu, 24 Aug 2023 05:31:25 +0000 (07:31 +0200)]
Continued:
- sorted domains for fetch_blocks after how many total blocks have already been
  found

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 11:25:03 +0000 (13:25 +0200)]
Continued:
- neodb is providing at least /apt/v1/instance/peers

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 07:47:13 +0000 (09:47 +0200)]
Continued:
- strip spaces off

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 07:10:40 +0000 (09:10 +0200)]
Continued:
- added "dynamic alias" for software type 'gnusocial'

14 months agoContinued:
Roland Häder [Wed, 23 Aug 2023 04:09:12 +0000 (06:09 +0200)]
Continued:
- added parameter --mode for updating node-info for detection mode
- this mode does only check outdated instances

14 months agoContinued:
Roland Häder [Tue, 22 Aug 2023 18:34:24 +0000 (20:34 +0200)]
Continued:
- added parameter --no-software which fetching instances with no software
  detected and not recently being checked
- the parameter --force is not re-checking recently entries. If you want this
  you need to use ./nodeinfo.sh --software --force but be kind to other
  webmasters!

14 months agoContinued:
Roland Häder [Fri, 18 Aug 2023 01:49:57 +0000 (03:49 +0200)]
Continued:
- some websites are so badly configured that they return a broken gzip body
- let's handle that exception here, too

14 months agoContinued:
Roland Häder [Thu, 17 Aug 2023 18:01:32 +0000 (20:01 +0200)]
Continued:
- wording fixed

14 months agoContinued:
Roland Häder [Thu, 17 Aug 2023 03:42:40 +0000 (05:42 +0200)]
Continued:
- ops, don't link None

14 months agoContinued:
Roland Häder [Wed, 16 Aug 2023 23:04:11 +0000 (01:04 +0200)]
Continued:
- added alias 'russkey' for 'misskey'

14 months agoContinued:
Roland Häder [Wed, 16 Aug 2023 22:56:19 +0000 (00:56 +0200)]
Continued:
- no, nope: validators.hostname() was a bad idea, it also let IP addresses and
  local host names in as well
- added command remove_invalid to remove those from database
- renamed recheck.sh -> nodeinfo.sh

14 months agoContinued:
Roland Häder [Wed, 16 Aug 2023 13:27:44 +0000 (15:27 +0200)]
Continued:
- chaos.social isn't part of oliphant, so it still requires being handled
  separately
- fetch software/origin from local database instead software from remote
  nodeinfo (saves some requests to their servers)

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 20:48:50 +0000 (22:48 +0200)]
Continued:
- links from infos.html view to list.html added

14 months agoFixed:
Roland Häder [Tue, 15 Aug 2023 20:24:41 +0000 (22:24 +0200)]
Fixed:
- I missed changing this one

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 20:09:54 +0000 (22:09 +0200)]
Continued:
- use validators.hostname() as these are host names

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 19:21:09 +0000 (21:21 +0200)]
Continued:
- fixed instance (host) names with two dots causing a UnicodeError

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 19:12:43 +0000 (21:12 +0200)]
Continued:
- need to skip recently but unknown instances (software is None but nodeinfo
  has recently being updated

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 19:05:10 +0000 (21:05 +0200)]
Continued:
- don't invoke federation.fetch_instances() when last_instance_fetch is
  recently being updated (means recently being fetch already)
- column for instances.is_recent() needs to now only start with "last_"
- don't determine unknown software if instances was recently fetched

14 months agoFixed:
Roland Häder [Tue, 15 Aug 2023 17:48:01 +0000 (19:48 +0200)]
Fixed:
- ops, didn't notice the inverted function name (my fault)
- also added proper debug message

14 months agoContinued:
Roland Häder [Tue, 15 Aug 2023 17:12:09 +0000 (19:12 +0200)]
Continued:
- typical for oliphant members: Hide their own blocklist and then handle it
  over to Codeberg repository of oliphant
- this helper now has a simple function to check if the provided domain should
  be excluded

14 months agoContinued:
Roland Häder [Mon, 14 Aug 2023 04:03:15 +0000 (06:03 +0200)]
Continued:
- added detection-mode 'APP_NAME' which reflects meta information
  name="application-name"
- allow checking generator type if status code 410 (Gone) is given, e.g.
  wordpress.com still returns a full HTML code to check

14 months agoContinued:
Roland Häder [Sun, 13 Aug 2023 18:20:11 +0000 (20:20 +0200)]
Continued:
- also re-check 'GENERATOR' detection mode

14 months agoContinued:
Roland Häder [Sun, 13 Aug 2023 18:03:59 +0000 (20:03 +0200)]
Continued:
- switched: PLATFORM before GENERATOR, the last one can be "personalized"

14 months agoContinued:
Roland Häder [Sun, 13 Aug 2023 17:13:36 +0000 (19:13 +0200)]
Continued:
- added --software2 for re-checking instances with `software` given and no
 `detection_mode` given
- also added og:platform to HTML base template

15 months agoContinued:
Roland Häder [Sun, 13 Aug 2023 14:50:12 +0000 (16:50 +0200)]
Continued:
- combine all discourse into one

15 months agoContinued:
Roland Häder [Sun, 13 Aug 2023 13:53:13 +0000 (15:53 +0200)]
Continued:
- no need to enter json.from_response() when response isn't okay or status code
  != 200 was found

15 months agoContinued:
Roland Häder [Sun, 13 Aug 2023 06:08:35 +0000 (08:08 +0200)]
Continued:
- show depth here, too

15 months agoContinued:
Roland Häder [Sat, 12 Aug 2023 21:31:09 +0000 (23:31 +0200)]
Continued:
- 'nube' is Spain and means cloud in English, this is another (valid) alias for 'nextcloud'

15 months agoContinued:
Roland Häder [Tue, 8 Aug 2023 23:33:16 +0000 (01:33 +0200)]
Continued:
- renamed report.txt to report.log

15 months agoContinued:
Roland Häder [Tue, 8 Aug 2023 20:03:52 +0000 (22:03 +0200)]
Continued:
- these commands can never have origin=None, so let's prevent them here

15 months agoContinued:
Roland Häder [Tue, 8 Aug 2023 19:48:58 +0000 (21:48 +0200)]
Continued:
- fba.commands.fetch_blocks() does the last checkup if the domain is valid et
  cetera, including deobfuscation

15 months agoContinued:
Roland Häder [Tue, 8 Aug 2023 18:14:54 +0000 (20:14 +0200)]
Continued:
- you can now with --feed=https://some-fba/feed.atom specify an other ATOM feed
  from an FBA/Pleroma bot
- parserset_defaults() is now specified first, then additional parameter

15 months agoContinued:
Roland Häder [Tue, 8 Aug 2023 12:59:40 +0000 (14:59 +0200)]
Continued:
- added __version__

15 months agoContinued:
Roland Häder [Mon, 7 Aug 2023 08:41:04 +0000 (10:41 +0200)]
Continued:
- is_recent() check doesn't make sense here, if the domain isn't registered
  it cannot be recently checked (ops)

15 months agoContinued:
Roland Häder [Sun, 6 Aug 2023 23:25:19 +0000 (01:25 +0200)]
Continued:
- also store peer count for relays

15 months agoContinued:
Roland Häder [Sun, 6 Aug 2023 21:40:58 +0000 (23:40 +0200)]
Continued:
- added debug line

15 months agoContinued:
Roland Häder [Sun, 6 Aug 2023 05:05:04 +0000 (07:05 +0200)]
Continued:
- ignore unsupported content type when response.text has zero length

15 months agoContinued:
Roland Häder [Sun, 6 Aug 2023 01:01:37 +0000 (03:01 +0200)]
Continued:
- ops, missed 1st parameter

15 months agoContinued:
Roland Häder [Sat, 5 Aug 2023 21:57:07 +0000 (23:57 +0200)]
Continued:
- added --timeout option for easy re-checking instances with timeout
- please increase "connection_timeout" and "read_timeout" config keys first!

15 months agoContinued:
Roland Häder [Sat, 5 Aug 2023 21:54:19 +0000 (23:54 +0200)]
Continued:
- added network 'mammuthus[ experimental]' for retriving peers
- moved software-related (not version number) functions to software.py
- strip off " experimental", so you can enter e.g. 'mammuthus' easier

15 months agoContinued:
Roland Häder [Sat, 5 Aug 2023 16:17:20 +0000 (18:17 +0200)]
Continued:
- don't link or show 'None', just a dash is okay

15 months agoContinued:
Roland Häder [Sat, 5 Aug 2023 13:52:13 +0000 (15:52 +0200)]
Continued:
- throw exception again, if they happen then they won't be fixed within a
  split of a second
- also make sure that home directory of FBA is properly set, sure you can
  choose a different directory or take the default /home/fba/
- added recheck.sh, a small wrapper script I wrote for myself and you should
  try. For example above exceptions might cause the used software not being
  detected (sure with timeouts) then you can run ./recheck.sh --software
  to re-test them

15 months agoContinued:
Roland Häder [Fri, 4 Aug 2023 08:36:36 +0000 (10:36 +0200)]
Continued:
- ignore 'None' as content-type it often happens with Lemmy instannces

15 months agoContinued:
Roland Häder [Thu, 3 Aug 2023 19:33:05 +0000 (21:33 +0200)]
Continued:
- update last_blocked timestamp when a blocklist has been fetched/generated for
  an instance aka. "blocker"

15 months agoContinued:
Roland Häder [Mon, 31 Jul 2023 23:10:27 +0000 (01:10 +0200)]
Continued:
- also mode=obfuscator is showing software names, let's link them, too

15 months agoContinued:
Roland Häder [Mon, 31 Jul 2023 09:51:56 +0000 (11:51 +0200)]
Continued:
- prevent blacklisted entries being processed, throw an exception if it happens

15 months agoContinued:
Roland Häder [Fri, 28 Jul 2023 08:19:39 +0000 (10:19 +0200)]
Continued:
- added re+ as alias but not with a regular + sign but UTF-8 encoded ...

15 months agoFixed:
Roland Häder [Fri, 28 Jul 2023 07:50:49 +0000 (09:50 +0200)]
Fixed:
- also need to check if data[json] has at least one entry
- mode=avg_peers shows software names, let's link to /list?mode=software here

15 months agoContinued:
Roland Häder [Thu, 27 Jul 2023 19:30:14 +0000 (21:30 +0200)]
Continued:
- don't reset during exceptions, maybe later they have fixed their instance

15 months agoContinued:
Roland Häder [Thu, 27 Jul 2023 10:59:53 +0000 (12:59 +0200)]
Continued:
- move nodeinfo handling to new module 'nodeinfo'
- also had to renamed variable nodeinfo to other names
- first newest version at /.well-known/x-nodeinfo2

15 months agoContinued:
Roland Häder [Thu, 27 Jul 2023 09:12:47 +0000 (11:12 +0200)]
Continued:
- check if bot_token is configured

15 months agoContinued:
Roland Häder [Thu, 27 Jul 2023 06:59:04 +0000 (08:59 +0200)]
Continued:
- again a lot of creativity was found, so let's clean this up to 'nextcloud'
  for later support (if they later add it)

15 months agoContinued:
Roland Häder [Thu, 27 Jul 2023 06:08:45 +0000 (08:08 +0200)]
Continued:
- added alias for forgejo

15 months agoContinued:
Roland Häder [Wed, 26 Jul 2023 20:24:12 +0000 (22:24 +0200)]
Continued:
- also show in scoreboard when an unsupported mode was provided
- fixed indenting

15 months agoContinued:
Roland Häder [Wed, 26 Jul 2023 15:49:52 +0000 (17:49 +0200)]
Continued:
- run.app is another mass-hoster for testing/developing code, no real instances
  are hosted there

15 months agoContinued:
Roland Häder [Wed, 26 Jul 2023 14:51:06 +0000 (16:51 +0200)]
Continued:
- command fetch_relays() should be locked
- fixed exception when row[column] is NoneType

15 months agoPartly reverted cdcd2b0109e126bca887d0712a7ddf602e5d6e62:
Roland Häder [Wed, 26 Jul 2023 14:22:34 +0000 (16:22 +0200)]
Partly reverted cdcd2b0109e126bca887d0712a7ddf602e5d6e62:
- "Accept" is not being accepted by misskey (gladly only these instances)
- it must be "Content-Type: application/json" or otherwise it is blocked

15 months agoContinued:
Roland Häder [Wed, 26 Jul 2023 10:34:05 +0000 (12:34 +0200)]
Continued:
- mode 'origin' added for header
- also unsupported are shown

15 months agoContinued:
Roland Häder [Wed, 26 Jul 2023 10:08:05 +0000 (12:08 +0200)]
Continued:
- moved/renamed utils.is_domain_wanted() to domain_helper.is_wanted()
- removed no longer used imports
- tpzo fixed: sof(t)ware

15 months agoContinued:
Roland Häder [Tue, 25 Jul 2023 18:29:21 +0000 (20:29 +0200)]
Continued:
- let also find instances by origin
- added target and title attributes

15 months agoContinued:
Roland Häder [Tue, 25 Jul 2023 12:17:48 +0000 (14:17 +0200)]
Continued:
- blacklisted 'vercel.app'

15 months agoContinued:
Roland Häder [Tue, 25 Jul 2023 02:34:30 +0000 (04:34 +0200)]
Continued:
- can be more accepted charsets

15 months agoContinued:
Roland Häder [Tue, 25 Jul 2023 02:19:04 +0000 (04:19 +0200)]
Continued:
- only set success (status code 200) when software was determined

15 months agoContinued:
Roland Häder [Tue, 25 Jul 2023 01:45:48 +0000 (03:45 +0200)]
Continued:
- owohub is yet another alias for misskey

15 months agoContinued:
Roland Häder [Tue, 25 Jul 2023 01:27:04 +0000 (03:27 +0200)]
Continued:
- request header is "Accept"
- also need to send api_headers (ops)
- need to convert variable 'path' to string

15 months agoContinued:
Roland Häder [Tue, 25 Jul 2023 00:44:31 +0000 (02:44 +0200)]
Continued:
- fixed instances.is_recent()
- blacklisted misskey-forkbomb.cf (the name says it)
- logger level changed to debug

15 months agoContinued:
Roland Häder [Mon, 24 Jul 2023 23:58:08 +0000 (01:58 +0200)]
Continued:
- added command fetch_joinmobilizon() to fetch instances JSON for mobilizon
  (federated events)

15 months agoContinued:
Roland Häder [Mon, 24 Jul 2023 22:51:40 +0000 (00:51 +0200)]
Continued:
- instances.is_recent() now checks recheck_block if 'last_blocked' is provided
- command fetch_blocks() now supports --force parameter
- blacklisted fnaf.stream as this domain has super-long sub-domains (troll)

15 months agoContinued:
Roland Häder [Mon, 24 Jul 2023 22:05:50 +0000 (00:05 +0200)]
Continued:
- ignorelist.com is also for testing/developing

15 months agoContinued:
Roland Häder [Mon, 24 Jul 2023 21:58:15 +0000 (23:58 +0200)]
Continued:
- added column `obfuscated_blocks` to save count of (still) obfuscated blocks
- also exposed it in infos.html view
- blacklisted gitpod.io as this domain floods `instances` table

15 months agoContinued:
Roland Häder [Mon, 24 Jul 2023 14:35:53 +0000 (16:35 +0200)]
Continued:
- added command fetch_relay() for fetching instances from ActivityPub relays
  which show their peers in index page (/)
- added grid.tf as this flooded a lot "testing/developing" sub domains