TAC Arrays, section managers

June 17, 2020

While porting from Python 2.7 on Google App Engine to Python 3.7 on Dreamhost, I ended up touching a solid percentage of my front-end JavaScript code. The changes started from integrating the generated persistent object reference manager, but then I added some new features and did some general UX streamlining. In the process, I revisited some of my general UI concepts, and figured I would write that up in case it might be useful to someone else.

TAC Arrays

A JavaScript web app needs to write HTML. For me the most natural way is using Tag-Attributes-Content (TAC) arrays. Here’s a simple example:

    var tac = ["div", {id:"maindiv"},
               [["p", {id:"firstp"}, "First paragraph text"],
                ["p", {id:"secondp"}, "Second paragrph text"]]];

As you can see the Tag is an HTML tag name, the Attributes are an object, and the Content is a string or another TAC array. A TAC array doesn’t argue with JavaScript syntax very much, and it’s straightforward to convert to HTML with a tac2html utility function.

Unfortunately HTML attributes like “class” and “for” need to be quoted (or abbreviated) so they con’t conflict with JavaScript reserved words, and hyphenated attributes like “data-state” will always have to be quoted. Aside from those exceptions, most common attributes can be used directly, which saves unnecessary quoting. Here’s an example from the membic source showing various attributes:

    return jt.tac2html(
        ["div", {cla:"revtseldiv", id:"revtseldiv" + cdx,
                 "data-state":"collapsed"},
         typemgr.typesHTML(cdx, mt)]); },

Connecting Events

For HTML to do anything, it needs to react to events. One way to do this is to write the HTML, then write code to connect events to it after it has rendered. There are times when that makes sense, but there are many more times where it adds complexity without much benefit. For simple cases, it is often clearer, and less error prone to write the HTML and the event connections directly in the TAC. For example here’s a typical “onclick”:

        ["a", {href:"#changetype", title:"Change Membic Type",
               onclick:jt.fs("app.membic.typesel('" + mt.type + ")")},
         typemgr.imgHTMLForType(cdx, mt)]);

In terms of writing and connecting HTML, this approach is relatively clear and easy to manage. The drawback is it requires exposing a “typesel” function in the public facing interface of the “membic” app module closure. That’s not inherently a problem, but it erodes the public facing interface in terms of module integration. And the code to write the HTML is being separated from the code handling the click event.

Code for UI subsection handling needs to be collected. Degradation of the public facing module interface should be avoided.

UI Section Managers

A UI section manager is an object within a module, providing functions for writing HTML and reacting to events. A section manager holds the code matching a logical section of the user interface.

Static event handler connections in HTML can be routed back to the section manager through a single public dispatch method at the module level. For example, something like this:

    managerDispatch: function (mgrname, fname, ...args) {
        switch(mgrname) {
        case "mymgr": return mymgr[fname].apply(app.mymodule, args);
        //other manager names here
        default: jt.log("mymodule.managerDispatch no manager: " + mgrname); }
    }

Routing all static event handlers through a single method might make the “onclick” HTML a bit longer to type, but that can be easily factored into a utility function in the section manager. For example

    function dispatchstr (mgrf, argstr)
        mgrf = mgrf.split(".");
        return "app.mymodule.managerDispatch('" + mgrf[0] + "','" + mgrf[1];
            "'," + argstr + ");return false;"
    }

Would then allow you to write a TAC array like

        ["a", {href:"#changetype", title:"Change Membic Type",
               onclick:dispatchstr("membic.typesel", "'" + mt.type + "'")},
         typemgr.imgHTMLForType(cdx, mt)]);

At the time of this writing, I’ve used this technique to factor out a theme posting manager, a detail fields manager, a link type manager, a rating manager, and a keyword manager. Still liking the approach.

Hope you find this technique helpful.

Advertisement

Porting Data from Google App Engine

March 5, 2020

Facing significant changes porting from Python 2.7 to 3.7, the time has come to move the membic project off Google App Engine. It might move back to GAE later, but it needs a clean port first, as the updates to persistent storage and caching are too significant to do incrementally on the running system. Fortunately, BigTable is the least featured database system I’ve ever used, so it should be possible to move from there to pretty much anything.

Exfiltrating My Data

Yet another reason to appreciate a REST architecture: being able to pull data out of a system that can no longer be modified. But the REST API doesn’t include sensitive user data, so to kick things off that had to be scraped from the database access console. Since there weren’t a crazy number of users, the fastest approach was to page through with a tedious copy-paste into a big file. Then some emacs replacement and macro operations to normalize into tab delimited format.

Armed with a uids.tdf file, the next step was to write a node.js program to walk the user data entries and make supporting membic API calls fetching all public information into separate json and image files. I loaded each object into a separate JSON file organized into subdirectories by object type. Associated image files were put in subdirectories off the object dirs. Even with everything separated, the async nature of node made progress hard to follow and things tended to stack up, so I added some sequencing to make it easier to follow. Probably took slightly longer, but it felt better watching the log file progress. I was worried about blowing access quotas for the day, but the App Engine server side caching (no longer available for free) meant I got through the whole data pull with no issues.

With all the data eventually pulled down to my development box, the next step was to port to the latest python version, and a new data store.

The Migration Platform

When I first migrated to App Engine, I switched from Java to Python, which turned out to be substantially more effective in terms of getting things done. Switching to WSGI and 3.7 will be an improvement. For data storage and retrieval, any reasonable database should be adequate in the near to medium term. My primary application hosting requirement is that the environment should take care of things like security monitoring, data backup, and general service patching.

After weighing other large service providers, a more traditional hosting plan with a flat monthly rate for service within specified levels was the best match. I’ve used Dreamhost for many years, and they offer Python with MySQL. I have a preference for Postgres, but I was pleased to learn MySQL has improved a lot over the years.

Aside from actually migrating the app, a major goal in this process is to absolutely minimize CRUD related code dependencies. Preferably without introducing another technology that itself creates impediments to future migration. I want to be able to handle my needs declaratively, with effects extending consistently from client side JavaScript, through server API calls, over to the data store, and back.

What that means is that after several years off, I’m back writing code that writes code. This time using JavaScript.

Generating CRUD Support

Declaring CRUD support starts with field description definitions. Here’s the descriptors that I wanted to work with:

var fieldDescriptors = [
    {dn:"priv[ate]", h:"authorized access only e.g. owner personal info"},
    {dn:"adm[in]", h:"administrative access only e.g. activation codes"},
    {dn:"req[uired]", h:"Save error if null or empty"},
    {dn:"uniq[ue]", h:"Indexed. Save err if matches another's value"},
    {dn:"str[ing]", h:"Rough max 128 char text, truncation ok.", aliases:[
        {dn:"email", h:"email address format"},
        {dn:"isod[ate]", h:"ISO date format"},
        {dn:"isomod", h:"ISO date;int count"}]},
    {dn:"text", h:"unindexable max 1mb string.", aliases:[
        {dn:"json", h:"JSON encoded data."},
        {dn:"idcsv", h:"comma separated unique integer ids"},
        {dn:"isodcsv", h:"comma separated ISO date values"},
        {dn:"gencsv", h:"general comma separated values"},
        {dn:"url", h:"a URL, possibly longer than 128chars"}]},
    {dn:"image", h:"base64 encoded binary image data (max 1mb)"},
    {dn:"dbid", h:"long int db id translated to string for JSON"},
    {dn:"int", h:"low range integer value JavaScript can handle"}];

YMMV. This is just what works for me. I’ve written that here as an illustration to show how field descriptions then allow me reasonably declare what I want to store, with comments. For example, here’s what my user entity looks like:

{entity:"MUser", descr:"Membic User account.", fields:[
    {f:"importid", d:"dbid unique", c:"previous id from import data"},
    {f:"email", d:"priv req unique email"},
    {f:"phash", d:"adm req string"},
    {f:"status", d:"priv string", c:"Only Active may post",
     enumvals:["Pending", "Active", "Inactive", "Unreachable"]},
    {f:"mailbounce", d:"adm isodcsv", c:"latest bounce first"},
    {f:"actsends", d:"adm gencsv", c:"latest first isod;emaddr vals"},
    {f:"actcode", d:"adm string", c:"account activation code"},
    {f:"altinmail", d:"priv unique email", c:"alt mail-in address"},
    {f:"name", d:"string", c:"optional but recommended public name"},
    {f:"aboutme", d:"text", c:"optional description, website link etc."},
    {f:"hashtag", d:"unique string", c:"personal theme direct access"},
    {f:"profpic", d:"image", c:"used for theme, and coop posts"},
    {f:"cliset", d:"json", c:"dict of client settings, see note 1"},
    {f:"themes", d:"json", c:"theme reference info, see note"},
    {f:"lastwrite", d:"isod", c:"latest membic/preb rebuild"},
    {f:"preb", d:"json", c:"membics for display w/opt overflow link"}],
  logflds: ["email", "name"]},

The logflds declares what fields to use for log messages. There’s also a queries field for things like my Membic entity, which requires indexes on multiple fields to support fast queries:

  queries: [{q:[{f:"ctmid"}, {f:"modified", dir:"desc"}]},
            {q:[{f:"ctmid"}, {f:"penid"}, {f:"modified", dir:"desc"}]}]},

With these declarations in place, I have everything I need for a makeMySQLCRUD.js file that generates

  • A createMySQLTables.sql file to initialize my database.
  • A dbacc.py file for use server-side, with entity CRUD, reference caching, marshalling of field values from request parameters through to database and back via JSON with images fetched separately, appropriate logging and error handling.
  • A refmgr.js file for use client-side, with locally cached fetch, automatic serialization/deserialization, and full access to details about fields and entity definitions.

Writing these files is not rocket science. It’s what you would expect. Even more importantly, it’s exactly what you need for your app, not for a general abstraction of an app that may or may not match your needs. If you want to use my generator as a reference, it’s at https://github.com/theriex/membic/blob/master/ref/port/makeMySQLCRUD.js

The point here is this approach works and it’s not even that hard. It’s cleaning a lot of crap out of my code as I move forward. HTH.

Python2 to Python3 on App Engine

January 23, 2020

With Python 2.7 at end of life, my hosting on Google App Engine (GAE) needs to migrate. This is not trivial:

  • webapp2 is replaced with WSGI.
  • The local development server is gone.
  • No more free caching.
  • All String database fields are indexed now.
  • Database image support is gone.

For me, the caching is of particular concern, because I was relying on that to smooth out db access. Hitting the db is fine, but if it happens a lot repeatedly, server side caching is what prevents exceeding db quotas. My reason for hosting on GAE was to put a free app out there, grow it past demo level with a handful of users, then get funding when the model is proved. Having the app already deployed on world class scalable infrastructure with zero technical impediments to growth was a selling point. Now it’s looking increasingly like it might not be worth it.

Porting a simple app

To check out the porting process, I converted a simple Python 2.7 webapp2 back-end with no database. Redoing my 4 API endpoints to Python 3.7 WSGI using the Flask framework was fairly painless, and an improvement. Running it locally was also not too hard, but it took me a few days to accept the idea that what I would be running locally would be completely separate from what I was running on GAE. I finally bit the bullet and installed nginx and gunicorn, connecting them and setting up the config to emulate my app.yaml declarations.

The general approach of moving utilities out of GAE in favor of standard Python modules makes a lot of sense, but is not always smooth. For example, moving from google.appengine.api urlfetch to urllib is straightforward, but a connection that had been working smoothly for years suddenly stopped because the default User-Agent: Python-urllib/3.7 appears to be persona non grata most places on the web. Of course where you are calling from can also make a difference, so this is always a question, but it’s indicative of the danger of equivalence assumptions for libraries when converting code.

Aside from that annoying issue, migrating my site was smooth, the code ended up a lot cleaner, and the result generally inspired confidence. If your database needs are minimal to non-existant, GAE is a solid option.

Porting with data

My other apps that I need to port store data. Most of the modules base their data access on google.appengine.ext db.Model, while making significant use of google.appengine.api for both images and memcache. I need to jump forward all the way to cloud NDB, and probably convert my existing data. I will need some other way to deal with images. I will need to write a cache interface to preserve all the time invested in logically solid server side caching, while not actually caching anything anymore. Then hope everything works without blowing quota in short order.

Daunting.

Given the magnitude of changes, I think my path forward is to migrate everything to an alternative host, leaving the existing site alone until the new one is ready. After switching, I can explore switching back.

Conclusions

When I first started this conversion process, it took me a while to find out what the path was going to be, and what changes affect viability. I put the summary list together in case it helps anyone else in a similar position.

When porting, I’m going to try factor my data handling so it only makes use of the highly constrained access provided by app engine. I’d like the option to move back, if the platform is viable.

Randomizing Content

January 20, 2020

Forgot I had a blog. Neglected to post the most recent rreplay album (Falling Through Frames) or any solo recordings that have happened in the meantime. Seems like my original thinking about documenting my music creation process isn’t really happening with any consistency. Meanwhile I’m finding I need a place to put other things as well.

Blogs are supposed to be related to a given topic so people following them can treat it like a channel. A more reasonable use might be to think of a blog as a place for saving writeups of things for later reference, or that might be useful to others searching for specific information. In other words a repository for generalized or condensed concepts. Alternately a place to store checkpoints in time related to significant directional decisions. Mostly for reference.

Topics are going to range a bit. Let’s see how that goes.

Vendredi

September 14, 2016

I hadn’t bought a vinyl record in a very long time, but having rreplay featured on Deviere’s latest release from Until My Heart Stops was a big deal for me.  When it finally went to press I picked up a copy which arrived a few days ago.  After dusting off my turntable and setting up to digitize (wanted this on my phone also) I rediscovered that vinyl is hot.  Like really hot.  Not only are the levels high enough that you will need to back it off significantly from your usual settings, but the sound really is noticeably different.  It’s warm.  This used to be my norm before digital, and wow.  If you haven’t listened to an album recently (by which I mean any recent album) you might want to do that.  I’d kind of forgotten.

Resolving to listen to vinyl significantly more frequently now.

Live From The Uncanny Valley

December 20, 2015

Eventually Slimpocket started to make some recordings, but publishing wasn’t that high a priority. Ultimately I became part of the revolving band personnel and moved back to Honolulu. While creating more material with rreplay, Rich pushed for making some forward progress on releasing some of our collected material. This culminated in the release of our second album amusingly and appropriately entitled “Live From The Uncanny Valley”. The title actually led to some interesting release processes with CD Baby deciding every track should have “live” appended to it in parentheses. There just isn’t much automatic infrastructure set up for improvisational electronica it would seem. In any case, you can get access to the album at cdbaby or amazon

Of course if the first album “rrepertoire” is somehow missing from your collection, you can still obtain a copy of that from cdbaby or amazon also.

Slimpocket update

May 15, 2011

So slimpocket is now well on its way towards completing an initial set
list. No spoilers, so here’s a couple of outtake jams in the interim.

03jam.mp3

09jam.mp3

Slimpocket

October 5, 2010

So I thought a blog would be an awesome way to kind of keep some
thoughts about the music process as it happens, but I haven’t been
doing well on updating it. At this point I am extremely late in
mentioning that I’ve connected with a resurrection of Slimpocket.
It’s kind of a mix of their existing material and new things that get
spontaneously created. Their existing recordings are probably still
best for the older stuff, but here’s a few jams that happened along
the way which I think are notable:

Riot:

Would You:

All Day Night:

Shocked:

June 28, 2009 jam

July 3, 2009

Notes from 28jun09 jam with Dave Savary and Matt Durso at the Sound
Museum 155 North Beacon St Brighton. Not much ventilation, but
overall I think the sweatbox approach was inspirational. Vocals
didn’t make it through very well. Perhaps next time I’ll put the
recorder somewhere not directly in front of the bass amp. Like maybe
by the door, closer to the PA. Maybe then it won’t be bouncing up
and down and recording the rattling ashtray/beercan whatever that was
next to it…

track01:
rocks hard. good development and starting to get some really nice
breaks.

track02:
rocks equally hard. Good development and dynamics. Recorder has some
automatic level setting which makes this hard to hear, but it was
definitely there.

track03:
more line composition so a bit slower to start. Actually like the
idea of turning this upside down occasionally if we do anything
similar to this again. Could edit to start at 2:05 or so.

track04:
Nice experimentation. Good change of pace. Really nice playing with
space. High level playing for a second jam.

track05:
Starting to get some pretty good subdivision on this. This group has
some good natural ability to pick up stuff quick. Meanders a wee bit
towards the end but some good ideas actually.

I thought we had more than that, but there’s no more on the recording.
I’m hoping it didn’t automatically shut off or something. I’ll try to
remember to check it whenever we come back from a break.

After a decade or so off, a rock jam

May 21, 2009

Jam with David Savary on guitar/voc and Matt on drums. Very rock, and much more enjoyable than I expected. Post rock?

track01: First time ever playing together, making it up as we go. I literally played two bars of the bassline, then we went for it.

track02: Starting from the guitar this time. I like what we came up with even if there might have been a few questionable notes. I seriously haven’t played anything like this in a decade at least.

track03: Again, two bars of bass omitted, but that’s it. A bit more space in this one. Like it. Fun making up bridges as we go along. And taking it out for a spin. Long jam and loses cohesion a little bit in there, but this is a process not a finished presentation. I actually would have liked to have let it fall apart even more completely if it was going to, if we play a bit more we’ll probably get more comfortable letting it crash and then picking it back up. We kick back in pretty well although we get a bit rambling after that.

track04: Somebody mentioned Morphine at some point, don’t know if it was before or after this jam. Cool feel though.

track05: Pardon the missing beginning, forgot to press the record button after we came back from a break. Glad I remembered even if I missed the first part of this. I don’t know how we ended up talking about terminal velocity of free fall. This just rocks.

track06: Nice atmosphere on this one. Piecing things together as a group.

track07: The first two minutes are a demonstration of figuring out something that isn’t going to work. Included that just to show that not everything worked out immediately. Then we hit the reset button and it sounded pretty awesome IMHO. You can tell where the volume changes a lot and the recorder adusts the eq to match. Use your imagination to pretend it gets lots quieter in spots.