Posts are no longer scanned for additional urls after recent API update

[AlexCSDev]

Recently patreon updated their API to represent post contents as json (content_json_string field) instead of html (content field, now gone). This broke additional url extraction using PluginManager.ExtractSupportedUrls because it expects html data.

In order to fix that PluginManager and plugins are likely need to be updated with additional enum input variable which will tell plugin what kind of input data is provided (HTML or Text) and PatreonPageCrawler will likely need to pre-process json to make a raw text out of it.

Update:

Quick glance at the json format shows that all links are possibly being provided as "type":"link" entries in the json, so there might be a better way of handling it - finding all of those entries in json and creating a new subentry with the type of PatreonCrawledUrlType.ExternalUrl for each of them might work. That will make previous call of PluginManager.ExtractSupportedUrls completely obsolete.

Please note that contributions created with the help of AI agents are not accepted.

[MathiusD]

Hi,

I occasionally work on another Patreon archiver in my spare time, so I've already had to look more closely at the content_json_string field. Currently, to retrieve links, I use the following fields:

• Image.attributes.media_id (If available)
• Image.attributes.src (If media_id isn't specified)
• CallToAction.attributes.button_link
• Text.marks
  • LinkMark.attributes.href

If you need to take a look, you can find my project here (In Crystal).

AFAIK, this field content_json_string follow the following rules :

Root element

Base type : Doc

Schemas

Basic

All schemas are following structure

{
  "type": "discriminatorValue"
  // Other attributes
}

Doc

{
  "type": "doc",
  "content": [
    // Any value following one the following schemas : Paragraph, Image (Only as block afaik), OrderedList, BulletList, Blockquote, Heading, CallToAction
  ]
}

Paragraph

{
  "type": "paragraph",
  "content": [
    // Any value following one the following schemas : Text, HardBreak, Image
  ],
  // optional
  "attrs": {
    "node_text_alignment": "left" // I've see only with following values : "left" & "center" atm
  }
}

Image

{
  "type": "image",
  "attrs": {
    "alignement": null, // I've only seen the following values : null, "full" & "center" (only with a concrete value in block variant atm)
    "alt": null, // Type string, optional (only with a concrete value in block variant atm)
    "caption": null, // Type string, optional (only with a concrete value in block variant atm)
    "height": null, // Type string, optional (only with a concrete value in inline variant atm)
    "width": null, // Type string, optional (only with a concrete value in inline variant atm)
    "link": null, // I've only seen with null value
    "media_id": null, // Type string, optional (can be null when media is external, eg: from discordapp)
    "src": "http://example.org/", // Type uri
    "node_type": "inline" // I've only seen inline or block
  }
}

OrderedList

{
  "type": "ordered-list",
  "content": [
    // Any value following one the following schemas : Paragraph, Heading, OrderedList, BulletList
  ],
  "attrs": {
    "order": 0 // I've only seen only positive integer on 8 bits
  }
}

BulletList

{
  "type": "bullet-list",
  "content": [
    // Any value following one the following schemas : Paragraph, Heading, OrderedList, BulletList
  ]
}

Blockquote

{
  "type": "blockquote",
  "content": [
    // Any value following one the following schemas : Paragraph
  ]
}

Heading

{
  "type": "heading",
  "content": [
    // Any value following one the following schemas : Text, Image (Only as inline afaik), HardBreak
  ],
  "attrs": {
    "level": 0 // I've only seen positive integer on 8 bits, so probably 1 to 6 (<h1> to <h6>)
  }
}

CallToAction

{
  "type": "cta",
  "attrs": {
    "fallback_strategy" : "fallback", // I've only seen this value atm : "fallback"
    "button_text" : "example", // type string
    "button_link" : "http://example.org/", // type uri
    "caption" : "example" // type string
  }
}

Text

{
  "type": "text",
  "text": "example", // type string
  "marks": [
    // Any value following one the following schemas : LinkMark, BoldMark, ItalicMark, UnderlineMark, StrikeMark
  ]
}

HardBreak

{
  "type": "hard-break",
  "marks": [
    // Any value following one the following schemas : LinkMark, BoldMark, ItalicMark, UnderlineMark (And probably StrikeMark, but i never hit this case atm)
  ]
}

LinkMark

{
  "type": "link",
  "attrs": {
    "href": "http://example.org", // type uri
    "target": null // I've only seen with following values : null, "_self" & "_blank" (Probably based on this one : https://developer.mozilla.org/fr/docs/Web/HTML/Reference/Elements/a#target)
  }
}

BoldMark

{
  "type": "bold"
}

ItalicMark

{
  "type": "italic"
}

UnderlineMark

{
  "type": "underline"
}

StrikeMark

{
  "type": "strike"
}

I hope my summary will be helpful. (I probably working on retrieval here if nobody work on it before and if I have some spare time during this weekend)

[AlexCSDev]

Hi, thank you for this information! It will certainly be helpful in the future.