API: get and filter multiple rems from knowledge base

Plugins and Custom CSS
#1

Using the frontend or backend API, is there a way to easily get all rem Ids of a knowledge base at once or even query the db to filter e.g. by tags or using regular expressions, etc.?

I’d like to filter my own knowledge base to get all rems with a specific tag so that I can further process this data in JavaScript. But all supported endpoints seem to only return one rem by a given identifier, not collections of rem.

I could use a search portal and query the ids from all rems of the search results (as they show up in the “visibleRemOnDocument” property), but I experience difficulties when searching for tags in search portals, results seem to vary and sometimes there are duplicates (maybe I’m doing something wrong).

It is also not easy to get the id of a search portal, since you cannot just copy the rem reference like with other rems, so it can only be retrieved by using the API to get the parent rem and copy the id from its children.

2 Likes
#2

API v0 has no capabilities to iterate all rem nor for advanced querying via regex etc. You can get all rem with a specific tag easily by loading all tagChildren that rem. Something like

await Promise.all(
  yourRem.tagChildren.map(
    async rId => RemNoteAPI.v0.get(rId)
  )
)

Making tons of requests is not a problem for the Frontend API since there are no actual requests sent to the server. It is all resolved locally/offline via IFrame message passing.

Here are some other approaches to get the more/all rems of you knowledge base, i.e. the rems of which tagChildren you want to list:

  • You could watch the children of the rem your plugin is in for any references/tags you’d type manually using the [[ search while the plugin is running, get their ID and fetch all tagChildren of those. The [[ search uses fuzzy matching so it satisfies at least some regex use cases. (If I understood you correctly this is similar to what you have done with the search portal, just completely manually?)
  • Iterate tagChildren of frequently used (power up) rem like Document, Header, Stub or Tag (enable Tag Collection in Settings, but it list only tags used after enabling this setting atm). Getting the name of all tagChildren of Tag and filtering them with regex sounds very much like what you want to do.
  • In theory you could structure your KB to be able to discover all rem by traversing the hierarchy root to leaf by following the children links. For this you just have to have every top-level rem tagged with a well known tag like #Document or #Stub.

If you just want to run some analytics for yourself you can also operate directly on the IndexedDB (like :gear: Smart Rem) by executing your code directly in DevTools. Or download a JSON backup (which is essentially just a dump of the IndexedDB) and process it with any means (e.g. Python remnote-explore/remexplore.ipynb at master · hannesfrank/remnote-explore · GitHub).

2 Likes
#3

Oh yes, getting the tagChildren should work, I didn’t think about that.

The traversing approach is something I’ve already been using, but in combination with tagChildren this could be much more powerful. Thanks for the suggestions!