The Frontside Podcast

How do we ensure a high level of quality and maximize the refactorability of our code? Frontsiders, Wil and Charles, talk about their battle tested techniques for testing web applications, not only in React JS, but in any JavaScript framework.

Links

Transcript

CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 90. My name is Charles Lowell, a developer here at The Frontside and your podcast host-in-training. And with me today is Mr. Wil Wilsman. Wil, who just got back from Nodevember just walked straight into the office and is ready to podcast with us on a very, very, very interesting subject, I think, today. We’re going to be talking about acceptance testing in JavaScript applications, especially some of the techniques that we’ve developed here around testing React applications based on the lessons that we learned from the Ember community. But really, more than just React applications. Really, testing any JavaScript application from the inside out, making acceptance tests for that.

So, I think we’re going to talk about some of the challenges that you encounter and some of the really novel solutions that are out there that we had nothing to do with. And I guess we really didn’t have, just more of cobbling together of various techniques for a powerful witch’s brew for acceptance testing.

Anyway, so Wil, just to round out the problem space or explore the problem space, what are some of the challenges that you encounter with an acceptance test? Actually, let me back it up even further. What is an acceptance test in a JavaScript application compared to what people normally encounter?

WIL: Acceptance testing or end-to-end testing is just a problem that every JavaScript app should face. Not everyone does, but they definitely should. And basically, it’s how the user interacts with your app through the browser. And every part of that we want to test, from the browser triggering browser events, interacting with the app, not calling functions or clicking buttons, and we’re pretending we’re a user.

CHARLES: Yeah. You know, I know that when we showed up in the React space, that was not really the way that most people tested their applications.

WIL: No, not at all. they’re all about unit testing. Make sure every small piece of your code works, and to some degree integration testing, making sure your components work with other components. But, nothing is out there really for those big acceptance tests that you want the user to click a button and expect them be brought to a page or these fields to be filled out, et cetera.

CHARLES: Mmhmm. Yeah. And there certainly was a very high level of maturity around unit testing, like you said. There are tools like Enzyme and…

WIL: Jest.

CHARLES: Yeah, Jest. But I was actually shocked to find out that Jest didn’t even run in the browser.

WIL: Yeah, it’s all virtual.

CHARLES: It’s all virtual. It’s completely and totally simulated and stubbed. And that presents some problems.

WIL: Yeah. The main problem is cross-browser testing. Some people might consider that to be separate from their acceptance testing but you should be able to just run your acceptance tests in multiple browsers and be able to also test cross-browser support.

CHARLES: Mmhmm. Yeah. And so, if you’re using something like Jest, you’re never actually running the code inside Safari. You’re never actually running it inside Internet Explorer. You’re actually running it in NodeJS.

WIL: And you know, your user is not going to run it in Node.

[Laughter]

WIL: They’re going to use a browser.

CHARLES: I don’t know about your users.

WIL: [Chuckles]

CHARLES: [Laughs] You know, we like to stick to the pretty advanced. It’s like, go to getNodeJSBrowser.com.

WIL: [Laughs]

CHARLES: Enough of this Firefox BS. But not seriously, it was certainly a problem. We were looking around, because we never like to build anything ourselves if we can avoid it. But it really just seemed like there was not an off-the-shelf solution for writing these big style acceptance tests in JavaScript. There are some services out there. There’s a couple now. What was the…

WIL: I think the main one here is Cypress.

CHARLES: Cypress, yeah. So, there’s Cypress now. I’ve watched the instructional videos but never actually tried to integrate it into my application.

WIL: Yeah. I think at its core it takes the same approach that we’ve been doing with how we’re interacting with our tests.

CHARLES: Mmhmm. Okay. The main difference is, is it that it’s a service? Like you have to edit your tests through…

WIL: Yeah.

CHARLES: Their web browser, their web interface, and use their assertion library?

WIL: Yeah. I’m not sure about the editing part. But yeah, it’s their assertion library. I’m pretty sure it’s their test runner and it’s their testing environment. Really, the only control through that is through their UI, or through settings, basically. And you’re stuck with those. You can’t use other… I don’t think you can use Mocha with Cypress.

CHARLES: Right.

WIL: Although it’s very much like Mocha…

CHARLES: Right.

WIL: It’s not.

CHARLES: Right, right. And I also noticed, we’ll touch on this later, the assertions, most of the side effects that were happening were happening right there inline inside your assertions. And that might be an opaque statement, but we will actually get into that later.

WIL: Yeah. And I think one of the things about their side effects so to speak is everything leading up to a side effect is a promise with Cypress.

CHARLES: Mmhmm.

WIL: So, when you select a button and click it, Cypress is going to wait for that button to actually exist before it clicks it.

CHARLES: Right, right, which is actually pretty cool. So, that’s actually a perfect into into one of the primary challenges with doing acceptance testing in general in a JavaScript application. This is a problem when you’re doing it in Ember. It’s a problem in React. It’s really a problem anywhere. And that is, how do you know when the effects of a user’s interaction have been realized? Right?

WIL: Yeah. And in Ember you take advantage of the run loop. Once that action happens, you wait for the run loop to complete and then your tests run.

CHARLES: Right. So, the idea is that I’ve clicked some button or I’ve typed some key or I’ve moved the mouse. And then I listen for the run loop and when it’s “settled” then I can now run my assertions because I know that the side effects that I was looking for have now been realized.

WIL: Yeah, hopefully, if you’re...

CHARLES: Hopefully.

WIL: Writing your app right. [Chuckles]

CHARLES: Right, right. But that actually presents some problems in itself because it requires visibility into the internals of the framework.

WIL: Yeah, so Ember is built with testing in mind.

CHARLES: Mmhmm.

WIL: And other libraries like React just being a view library might not be built with testing in mind. So, we don’t have those hooks to wait for this loop to complete, wait for all of these things to be rendered before you continue.

CHARLES: Exactly. And so, this is, I think it’s actually kind of both a blessing and a curse. Because there are such strong conventions in Ember, they were able to build this wonderful acceptance testing regimen from the get go.

WIL: Yeah.

CHARLES: But like you said, that doesn’t exist at all in the React ecosystem. And so, what do you do? There’s no run loop. You’re cobbling together a bunch of different components. And maybe you’re using Redux, maybe you’re using MobX, maybe you’re using… you’re certainly using React. And all of these things have their own asynchronies built-in. And there’s not one unifying abstraction that’s keeping track of all the asynchrony in the system. And so that presents a challenge. So, the question is then, if you’re trying to not actually check and observe the state of a system until the right moment, how do you know when that right moment is?

WIL: Yeah. And in early testing of a side React project I had, I would basically wait for a state to be complete before I continued my ‘before each’. And in the testing we’re doing now, it’s essentially what we’re doing except the state is what the browser sees, or what the user would see in the browser.

CHARLES: So you were actually querying the...

WIL: Yeah. So, I was using Redux. So, in my app I was saying, when the Redux app is done loading...

CHARLES: Mmhmm.

WIL: The instance is set to true or false, then continue the test.

CHARLES: Mmhmm. And so, what that means, what we’re doing, is doing the same thing except observing at the DOM...

WIL: Yeah, exactly.

CHARLES: Level. And what it means is we actually… I would love to set this up and have a big reveal but I guess we’ll just have a big reveal, is that essentially what we do is polling.

WIL: Yeah.

CHARLES: Right? So, when we run an assertion, let’s say you click a button and you want the button to become disabled, there’s an inherent asynchrony there. But what we will do is we’ll actually run the assertion to see if it’s disabled not one time. We’ll run it a thousand times.

WIL: Yeah, as many times as needed until it passes.

CHARLES: Right, exactly. As many times as needed until it passes. And I think that is, at least to most programmer instincts, an odious idea.

WIL: Yeah.

CHARLES: [Laughs]

WIL: It’s like, “Oh wait, you’re just looping over every single assertion how many times?”

CHARLES: Yeah, exactly. And it feels, yeah, it feels weird as an idea. But when you actually see the code that it produces, it just sweeps away so much complexity.

WIL: Yeah. And…

CHARLES: Because you don’t worry about asynchrony at all.

WIL: Yeah. And it’s pretty genius. If I’m a user and I click a button, it’s loading when I see that it’s loading. So, our tests are going to wait until that button says it’s loading. And then the test passes.

CHARLES: Right. And so, what we do is we essentially, we use Mocha but you could do it with QUnit or anything else, is that when you run your assertion, you declare, you have an ‘it’ block or I guess, what would it be in QUnit?

WIL: A test?

CHARLES: A test?

WIL: I think it’s just a test.

CHARLES: You have your test block. And so that function that actually runs the assertion and checks the state will actually run, yeah it could run three times. It could run a thousand times. It’s just sitting there waiting. And it will time out. And it will only fail if that assertion has failed a thousand times or it has failed through, I think two seconds is our default.

WIL: Yeah, yeah. I think we default to the runner’s default timeout.

CHARLES: To the runner’s default timeout, yeah.

WIL: Yeah. Or you can set that yourself with how we have it set up. And the other thing that comes from that is if your tests are only failing when they time out, how do you know what’s actually failing? And our solution to that was we catch the error every time it fails and right before the timeout actually happens we throw the real error.

CHARLES: Yeah. Exactly. But the net effect is that you’re able to write your assertions completely and totally oblivious of asynchrony. You don’t, we don’t have to worry about asynchrony pretty much at all. I mean, we do, and we’ll get into that. So, I made a global statement and then immediately contradicted it.

WIL: [Chuckles]

CHARLES: But hey, you got to be controversial. But for the most part, asynchrony just disappears because asynchrony is baked into the fabric. So, rather than thinking about it as a one-off concern or a onesie-twosie, it’s just every single assertion is just assumed to be asynchronous. And so, that actually means you don’t have to deal with promises. You don’t have to deal with run loops. You don’t have to deal with anything. You just write your assertion and when it passes, it passes. And there are some really unique benefits for this. And there are some challenges. So, I think one of the first benefits is that it’s actually way faster.

WIL: Right.

CHARLES: Which is counterintuitive.

WIL: It’s incredibly fast.

CHARLES: It’s very fast.

WIL: Yeah, for all the loops that’s happening you might think every loop is going to slow it down slightly. But it really doesn’t. Our tests, each test, even though it asserts five or six times, it takes milliseconds.

CHARLES: Mmhmm.

WIL: The test itself might only loop twice.

CHARLES: Right. Exactly. Whereas if you’re waiting for a run loop to settle, you might have some… you click a button, it disables, it also fires off an Ajax request and does all this stuff. But if all my assertion wants to know is “Is this button disabled?” then I only need to assert until that has happened. I don’t need to wait until all the side effects have settled...

WIL: Mmhmm.

CHARLES: And then do the assertion. I just know, “Hey, my assertion, the thing that I was waiting for - that happened. Let’s move on.” Yeah. And so, it’s so, so fast. And that was actually, I didn’t predict that. But I was definitely pleasantly surprised.

WIL: Yeah, that was a very nice surprise. And all of our tests ran so much quicker than they would have in a run loop environment with Ember or something.

CHARLES: Right. Yeah, yeah. That was, we actually had just come off a project where we were having that thing, that exact problem, which was that yeah, our animations were slow. Or, the animations were fine. They were perfect. [Laughs] But there were slowing down the tests.

WIL: Yes. I think in that project, it was like, 30-minute tests...

CHARLES: Mmhmm

WIL: For the whole suite to run.

CHARLES: Yeah. To which I’ll add a public service announcement. I think this is a conjecture but I do believe that animations are best applied not to individual components but by the thing that uses a component. So, I shouldn’t have an animation that’s like, implicit to a dialog. It should be the thing that’s showing the dialog that gets to decide the animation to use. Anyway, just throwing that out there.

WIL: [Laughs]

CHARLES: Because animations are about context. And so, the context should provide the animation, not the individual atom. Anyway, moving on.

WIL: Some other podcast.

CHARLES: Yeah.

[Laughter]

CHARLES: That’s another podcast right there. But there’s also, this does present some challenges or requires code to be structured in a way that facilitates this. So, there are some challenges with this approach, some things you need to be aware of if you’re using this kind of system. We’ve kind of settled on a name for what we call these types of assertions and these types of systems.

WIL: Yeah. We call them convergent assertions, because you’re converging on something to happen. It’s going over and over until it happens.

CHARLES: Right.

WIL: And yeah, a lot of these challenges that we’ve come across are things that you might not think of, like there are a few instances of false positives...

CHARLES: Mmhmm.

WIL: That happen with these convergent assertions.

CHARLES: Right. So, what would be an example there?

WIL: So, the most common example that I’m seeing so far is when you’re asserting that something didn’t happen.

CHARLES: Mm, right.

WIL: That would immediately pass. But if it takes your app...

CHARLES: [Laughs]

WIL: A few seconds for it to actually happen, then you could still have an actual failure but your test passed immediately.

CHARLES: Right, right. So, what’s the countermeasure then?

WIL: We invert our assertions. So, we make sure they fail for a certain amount of time.

CHARLES: Right. So, the normal case where you just want to say, “I want to make sure that my state converges to this particular state.”

WIL: Alright. I said fail at first. I meant, pass. We have to make sure it passes for a certain period of time.

CHARLES: Right, exactly.

WIL: So yeah, the normal way is it fails until it passes, and then it passes. When you invert one of these convergent assertions, you’re just making sure it passes repeatedly and if it fails at any point, you throw a failure.

CHARLES: Right, okay. And so, that’s like, if I want to check that the button is not disabled, I need to check again and again and again and again.

WIL: Until you’re comfortable with saying, “Alright. It’s probably not going to be disabled.”

CHARLES: Yeah, exactly. And so there, it’s kind of weird because it is dependent on a timeout.

WIL: Mmhmm.

CHARLES: You could go for two seconds and then at the very end it becomes disabled. So, you just kind of have to take that on faith. But...

WIL: Yeah.

CHARLES: In practice, I don’t think that’s been much of a problem.

WIL: No.

CHARLES: It’s more indicative of, if your button disables after...

WIL: A few seconds.

CHARLES: A few seconds, what’s up with your...

WIL: Yeah, what’s up with your app?

CHARLES: Yeah, exactly.

WIL: If you’re waiting for an Ajax request or something, an example, then you should be using something like Mirage Server.

CHARLES: Right. Which is, man, we got to get into that, too. There are a couple of other things that I wanted to talk about too, with these convergent assertions. And that is, typically when you look through the READMEs for most testing frameworks, you see the simple case of the entire test, the setup, the teardown, and the actual assertions, are in the actual test.

WIL: Yeah. The ‘it’ block in Mocha or the test block in QUnit. You click a button, and then make sure it’s disabled, and it moves onto the next test.

CHARLES: Mmhmm.

WIL: Then you click a different button or the same button and you assert something else in the next test.

CHARLES: Mmhmm. Right.

WIL: And yeah, you can’t do that with convergent assertions because they’re looping. So, if you click a button in a loop it’s going to keep clicking that button over and over and over again.

CHARLES: Yeah. [Laughs] Right, right. So, it means that you need to be very conscientious about separating the parts of your tests that actually do the things that actually act the part of the user from the part of your test that’s about observation.

WIL: Yeah. So, our solution to that is we move out all of our things that have side effects like clicking a button or filling in a form, all that stuff happens in ‘before each’s. And all of our actual assertions happen in these convergent ‘it’ blocks that loop over and over again. So, our ‘before each’ runs and clicks the button and then we have 10 or so tests that will loop and wait for various states to...

CHARLES: Yeah.

WIL: Be true.

CHARLES: Right. That means that yeah, all these assertions do is they read state. And you just have to, you do have to be conscientious. You’re not allowed to have any side effects inside your tests, your actual assertion blocks.

WIL: Mmhmm.

CHARLES: But that’s actually, it’s a good use case for the whole Act-Arrange-Assert, which has been around way, way, way before these techniques. But here, we’re doing Act and Arrange in our ‘before each’...

WIL: Mmhmm.

CHARLES: And then we’re doing Assert later. And I think it actually leads for more readable...

WIL: Yeah, definitely.

CHARLES: Things.

WIL: And it also opens the door to something that we can’t really take advantage of yet but if you have 10 assertions with one ‘before each’ side effect, you could run all of those assertions in parallel.

CHARLES: That’s right.

WIL: And your tests would be 10 times more faster.

CHARLES: Mmhmm. Yeah, exactly. Or you could run them in parallel or you could just run them one after the other but you wouldn’t have to run that ‘before each’ 10 times.

WIL: Yeah. But something with that, that I found, is if we move all of our side effects to ‘before’ blocks instead of ‘before each’ blocks, sometimes a test three tests down that’s waiting for something to happen...

CHARLES: Yeah.

WIL: That thing might have already happened earlier...

CHARLES: Yeah.

WIL: And it already went away. A loading state is the best example of that.

CHARLES: Mmhmm.

WIL: You show the loading state, the loading state goes away. So, if you move that button click into a ‘before’ and that loading state test is three tests in, that loading state is already going to be gone.

CHARLES: Yeah, so I think the long story short is we’ve kind of come to the conclusion that we would have to write our own runner.

WIL: Yeah.

CHARLES: Essentially to take advantage of this. But that said, we’ve done some sketching about what we would gain by writing our own runner. And the speed, we’re talking about exponential speedups.

WIL: Yeah.

CHARLES: Maybe taking an entire acceptance test suite and having it run in five or six seconds.

WIL: Yeah. We’re talking about these tests that are already extremely fast.

CHARLES: Mmhmm.

WIL: Each test takes a few milliseconds or tens of milliseconds to complete. But then if you can run all of those at the same time, all of your tests for that entire ‘describe’ block just ran tens of milliseconds.

CHARLES: Right. Yeah. So, it’s really exciting and pretty tantalizing. And we would love to invest the time in that. I’ve always wanted to write our own test runner. But never had, [chuckles] never really had a reason.

WIL: Yeah.

CHARLES: Certainly not just for the sheer joy of it. Although I’m sure there is joy in writing it. But that, yeah, we’ll have to wait on that. But I am actually really excited about the idea of being able to maybe bring this back to the Ember community.

WIL: Yeah.

CHARLES: Because acceptance tests getting out of control in terms of the speed is I think a problem with Ember applications. And I think this would do a lot to address that.

WIL: Yeah.

CHARLES: I think, how long, if we were just using a stock Ember acceptance testing setup for this, I think we have about 250 tests in this React app...

WIL: Yeah.

CHARLES: How long does it take to run?

WIL: Right now I think our tests take something like 20 seconds. And that’s also somewhat due to they have to print the tests on the screen on Travis so that takes a little time. In an Ember setup, that could maybe take a few minutes. I mean, that’s not that big of a deal, a few minutes.

CHARLES: Right.

WIL: But compared to 20 seconds.

CHARLES: Right. you’re still talking about an order of magnitude...

WIL: Yeah, exactly.

CHARLES: Difference. And using this, I think you could get a 30-minute test suite...

WIL: Yeah.

CHARLES: Down to the order of 3 minutes.

WIL: Now when we’re talking about those times, we’re talking about the tests themselves. Of course, the CI would have to download stuff and set up the [inaudible].

CHARLES: Mmhmm, right, right.

WIL: And that of course all adds to the time.

CHARLES: Yeah, mmhmm, yeah. Earlier you mentioned, we talked about Ember CLI Mirage. This is actually something that is having now been using it for what, 2 years or something like that, it’s just… it’s impossible...

WIL: To go back, yeah.

CHARLES: To go back. It is. It’s like [chuckles] you come outside the Ember community and you’re like, “How is anybody ever dealing without this?”

WIL: Yeah.

CHARLES: [Laughs]

WIL: A lot of the mocks are usually mocking the function that makes the request and it returns it in that function. That’s what’s out there currently, minus the Mirage stuff.

CHARLES: Mmhmm.

WIL: But once you use Mirage, you’re mocking the requests themselves.

CHARLES: Yeah. And you’ve got such great support for the whole factories. I love factories. It’s something that is very prevalent in the Ruby community, and maybe not so much elsewhere. But the ability to very, very quickly crank out high-fidelity production data...

WIL: Yeah. And you don’t have to have files upon files of fixtures.

CHARLES: Yeah, exactly. And you can change, if something about your schema changes, you can change the factory and now your test data is up and running. So, Ember has this tool called Mirage which is just, like I said, it’s so fantastic. Oh yeah, it’s also go support for running your application. Not just in your tests, but you can actually run your application with Mirage on and...

WIL: Oh right, yeah.

CHARLES: And you’ve got now the most incredible rapid prototyping tool.

WIL: Yeah. You don’t need to connect to a server to see fake data.

CHARLES: Right, right. And we were even talking about this yesterday to a potential client. they’re trying to, they’ve got to present something to investors. And how wonderful is it to just be like, “You know what? We just don’t want to invest in all, we don’t want to move the inertia, invest the money to generate the force to move the inertia of a backend.” Especially in this particular use case, the backend was going to be really, really heavy.

WIL: Yeah. And there were some questions about the backend that we couldn’t address quite yet but we wanted to start working on something that we could show, something demo-able.

CHARLES: Right, exactly. And so, Mirage is just so wonderful for that. But again, Mirage is, it’s an Ember-specific project.

WIL: Right.

CHARLES: So, the question was, “How are we going to use that?”

WIL: And you actually took this on yourself.

CHARLES: Mmhmm.

WIL: I just saw this pop-up one day and boom, you converted Ember Mirage to vanilla JavaScript.

[Chuckles]

CHARLES: So, I did extract it. But the lion’s share of the credit goes to the developers of Mirage themselves. Sam Selikoff and the Mirage community, they built Mirage not using much of Ember. There were some utilities that they were using, but mainly things like string helpers to convert between camel-case and dash-case, and using a Broccoli build, or using an Ember CLI build.

WIL: Yeah. That was one of the challenges that we came across using Mirage outside of Ember, was how do we autoload this Mirage folder with all this Mirage config and Mirage factories and models, et cetera.

CHARLES: Right. The internals were all just straight up JavaScript classes, for the most part. And so, extracting it, it was a lot of work. But 90% of the work was already done. It only took three or four days to do it.

WIL: Amazing.

CHARLES: Yeah. So, it was actually a really pleasant experience. I was able to swap out all of the Ember string helpers for Lodash. So now, it’s good to go. It shares a Git history with Ember CLI Mirage, so it’s basically a fork.

WIL: Mmhmm.

CHARLES: Like, a very heavily patched Ember CLI Mirage. But I keep it up-to-date so that it doesn’t...

WIL: Good. [Inaudible]

CHARLES: Yeah, so I think the last time I merged in from master was about a month ago, something like that. Because it’s got all the features that we need but it’s not a big deal to rebase or just to merge it on over in. Because yeah, it’s a really straightforward set of patches.

WIL: Was there any talk with the creators of Ember Mirage about getting this upstream?

CHARLES: So, I’ve talked a little bit with Sam about it. And from what I can tell, his feeling on it is like, “Hey, my goal right now is to focus on this being the best testing and data stubbing platform for Ember. Anything that happens out there, outside of that scope, that’s great. And I certainly won’t get in the way of it. But I’m pretty maxed out in terms of the open source credits that I have to spend.”

WIL: [Chuckles]

CHARLES: And there hasn’t been much motion there. I’m happy where it is right now. I would like to see it merged into upstream. I think it would be great to have basically this Mirage Server and then have Ember CLI bindings for it.

WIL: Yeah, yeah. I was going to say, either another Ember CLI specific package for Mirage or maybe to make it a non-breaking change or something.

CHARLES: Yeah.

WIL: Just like an Ember-specific entry point.

CHARLES: Right, exactly. And I think that’s definitely doable, if someone wants to take it on. I will say, we have been using this extracted plain vanilla JavaScript Mirage Server now for what, almost six months?

WIL: Yeah.

CHARLES: And it really hasn’t...

WIL: Yeah, I don’t think I ran into one issue with that.

CHARLES: Yeah. It’s solid. It’s really, really good. So, kudos to the Mirage team for doing that. And if anybody is interested in using Mirage in their projects, it’s definitely there and we’ll put it in the show notes.

WIL: Yeah. We call it Mirage Server.

CHARLES: Mirage Server, yeah. So, I don’t know. Maybe it’s time to reopen that conversation. But it has become a very integral and critical piece of the way that we test our JavaScript applications now.

So, what are the foundations of it? We’ve got, we’re using Mirage. We’re using these convergent assertions. We’re using Mocha, although that’s really...

WIL: Yeah, we have jQuery and Chai jQuery just to help us out with interacting with the browser as a user would. And I think one of the big challenges with that actually, I just remembered, was triggering changes in React.

CHARLES: Yeah.

WIL: I think this is pretty specific to React. You might run into problems with the view. I don’t know how to mess with view. But in React, at least I think 15 or React 16, one of them, they changed the descriptor of the value property on an element so that they can appropriately interact with it, make changes, watch for changes, et cetera. So, when you set this value property using jQuery or just straight up ‘.value()’, that change event isn’t triggered in React. Your on-change handlers are never called.

CHARLES: Wait, they actually update the JavaScript property descriptor of the DOM element.

WIL: Yes.

CHARLES: Boo.

WIL: Yeah. So, there’s a nice little helper out there called React Trigger Change. I dug through it and I’ve stripped some of it down to just be for more modern browsers, more modern React. But there’s a lot of good code in there. And basically, it just caches that descriptor, updates the value, triggers the change, and then adds the descriptor back. And that ends up triggering that React element.

CHARLES: Okay. [Laughs]

WIL: [Chuckles] Yeah, so that calls your handlers and that’s how we get around that.

CHARLES: Right. There’s a few fun little hacks there. But I think it is good to tie that into a larger point, is that the amount of touch that you have with the framework is actually very low.

WIL: Yeah.

CHARLES: So, the amount of affordances that we’ve had to make just for React, there’s that, that you just mentioned. And is there… there’s not much else. We had to write a test harness to mount the app. But that’s like...

WIL: Yeah, yeah. Our describe application helper is pretty React-specific.

CHARLES: Right.

WIL: So, you’d have to render it and set up a Mirage server, et cetera.

CHARLES: Right. But that’s application-specific setup.

WIL: Yeah, that’s one file.

CHARLES: Right.

WIL: So, your goal for acceptance tests is you want to be able to have a refactor and your acceptance tests still pass.

CHARLES: Right.

WIL: So, what if that refactor involves switching libraries?

CHARLES: Right.

WIL: If you’re writing Ember acceptance tests, you’re going to have to rewrite all your acceptance tests.

CHARLES: Right.

WIL: That’s a huge downside.

CHARLES: Right.

WIL: So, with this method of interacting with the actual library very little, we have that one file that sets up our app and then we have that one trigger change helper, we remove those, we can use whatever framework we want underneath this. And our tests would still work.

CHARLES: Yeah, exactly. And I think that we actually could theoretically, and honestly I have enough confidence in this style that we’re developing the tests now, we could refactor this application to Ember and not have to rewrite our tests.

WIL: Yeah, exactly.

CHARLES: In fact, the tests would be an aide to do that.

WIL: Yeah, and the tests would be faster than Ember testing with that run loop problem.

CHARLES: Yeah, exactly. that’s really something to think about or to think on, is like, “Wow. You’re really at this point completely, not completely, but very loosely coupled to the actual internal library code.” Which is one of the goals of a nice, big acceptance test, is to be able to make major changes, break big bones, and be able to set them and have your acceptance test suite be the bulwark that holds it all together.

WIL: Yeah.

CHARLES: So, I actually don’t know what a bulwark is.

WIL: [Laughs]

CHARLES: I just know that it’s a really strong thing.

[Laughter]

CHARLES: Maybe we could put that in the show notes.

[Laughter]

WIL: A link to what that is.

CHARLES: [Laughs] So, alright. Well, I’m trying to think if there is anything else that we wanted to mention. Any challenges? Any next things?

WIL: So, one of our next steps is something we mentioned that Cypress does, is they wait for elements to exist before they interact with them. And we’re actually not doing that in our app currently. And we don’t have helpers out there for it yet.

CHARLES: Right.

WIL: But that’s very much the next step. When we go to click an element in our ‘before each’, we have these describes that are nested. Say, you have nested describes and you get down three levels into a ‘before each’ when you’re clicking a button. That button might not exist yet.

CHARLES: Right.

WIL: And especially since we’re using jQuery, if you trigger a change on an empty jQuery element, it’s not going to throw an error. It’s just not going to tell you that it triggered anything.

CHARLES: Right.

WIL: So, we get those skips where that button’s not getting clicked and we should really be waiting for that button to exist.

CHARLES: Right. So, what we’ve done right now is we’re converging on our assertions at the backend of a test. But at the frontend of a test we need to also be converging at some state before we can actually interact with the application.

WIL: Right, yeah.

CHARLES: So yeah, so that part is missing. And that actually brings up, we are very slowly but nevertheless doing, we’re collecting these convergent assertions and convergent helpers in a repository on our GitHub account. We’re going to be adding these things so that you can either use them out of the box or use them to make your own testing library.

WIL: Yeah. And one of the other next steps that goes along with waiting for the element to exist is when you need to chain convergences. Like, wait for this element to exist and then click it and then wait for this thing to happen before actually running a test. And that presents the problem of our convergences are waiting for that timeout and those timeouts will accumulate. So if you have three chained convergences, that’s now a six thousand millisecond timeout as opposed to a two thousand millisecond timeout.

CHARLES: Right.

WIL: So, one of the next steps is getting that tracking under control so if you chain three convergences together, they’re smart about it and they still fail under the two thousand millisecond timeout.

CHARLES: Right, right. So yeah, so we’re going to be collecting all this stuff that we’re learning into some publicly available code. We have a repository set up. I don’t know if I want to announce it just yet, because it’s really early days.

WIL: Yeah.

CHARLES: But that definitely is the plan. And that way, whether you’re using Mocha or whether you’re using QUnit or whether you’re using Chai or jQuery, you’ve got these underlying primitives that help you converge on a state, whether that state is to interact with some piece of the DOM or to just assert some observation is made about that state. We’ll be continuing on that. But by all means, get in touch if this is something that is of interest to you. Let’s make something happen, because it’s something that we’re pretty excited about. And honestly, it’s pretty comfortable living inside the four walls of this test suite.

WIL: Yeah.

CHARLES: It feels pretty good.

WIL: It does, yeah. They’re very fast. And some places in the test might need a little reworking, but for the most part all of our tests are very well-written, very well-readable. And you can just open up a test and know exactly what’s going on.

CHARLES: Yeah. Alright. Well, I think that about does it for Episode 90. Wow, Episode 90.

WIL: Man, coming up on that 100.

CHARLES: Yeah. We’re going to have to have a birthday cake or something.

WIL: Do we celebrate Episode 100 or Episode 104?

CHARLES: What’s 104?

WIL: 104 would be 2 years.

CHARLES: Oh really?

WIL: Well, I mean 2 years’ worth of podcasts.

CHARLES: Oh, right. 2 years’ worth of podcasts. Yeah.

WIL: Yeah, like if you go every week.

CHARLES: Maybe we should celebrate a hundred hours or something like that.

WIL: Oh yeah.

CHARLES: We can add up the thing or celebrate… I don’t know, be like, “You’ve literally wasted 2 years of your life.”

WIL: [Laughs]

CHARLES: “2 weeks of your life listening to the podcast.” Anyway, so that’s it for Episode 90. and thank you so much, Wil.

WIL: Thanks for having me.

CHARLES: It’s always a pleasure to talk about these topics with you. And as always, if you need to get in touch with us, please reach out to us on Twitter. We are @TheFrontside. Or you can send an email to [email protected].

Toran Billups: @toranb | GitHub | Blog

Toran Billips joined us for an insightful conversation regarding glimmer-redux: Predictable state management for Glimmer apps.

Resources:

Transcript

ELRICK: Hello and welcome to another Frontside Podcast, Episode 89. My name is Elrick Ryan, a developer here at the Frontside. I'm joined by Wil Wilsman, another developer here at the Frontside. Wil, how are you doing?

WIL: I'm good. How are you?

ELRICK: I'm great, man. I'm excited for this podcast that we have coming up here. Today we are fortunate to have with us a podcast elite member now, Mr Toran Billups. Toran, how are you doing?

TORAN: Oh, man. I joined the elite platinum club or something?

ELRICK: Yes, you are in the platinum club right now. I think this is probably what? Your third or fourth episode by now?

TORAN: Oh, yeah. I think the fourth.

ELRICK: Oh, yeah. You're in the elite club right now. You are a Midwest programmer and I hear there is a difference between a Silicon Valley programmer and a Midwest programmer. Could you tell us about what the difference is? Because it's the first time I've heard anything about this.

TORAN: Admittedly, I stole this from a very popular developer, Justin Searls who spoke at length one time on a podcast, not too different in this one about his experiences in consulting for companies, who are more in the startup phase or a company that you'll find in Silicon Valley that is mostly just trying to test an idea and get to market, versus his experience for finance or insurance companies based out of the Midwest. I like that idea because my experiences have taught me. I'm a little bit happier when I'm working for companies that are interested in quality or attributes of quality and view the software longevity as mission-critical versus a software that is really just a byproduct of an interesting idea and if we validate that idea in a market, we can always rewrite the software later. Midwest, I guess the short version is we care about the work we're doing and we understand that rewrites are difficult, if ever possible.

ELRICK: Interesting, so the Midwest seems to be concerned with long term goals.

TORAN: Yeah, I think sustainable --

ELRICK: Sustainable software, at least.

TORAN: Yep.

ELRICK: Today, you are joining us to not only talk about the Midwest and the beautiful Midwest programmers. You're here to talk about Glimmer Redux.

TORAN: Glimmer Redux is a little library I wrote, I think last month. I should start off by asking you guys if you're familiar with a Glimmer JS or if you've heard of that.

ELRICK: I've heard of Glimmer JS. I haven't had an opportunity to play around or mess around with it yet. I don't know if that's good or bad because I'm just really busy but I really want to get into it. Wil, what about yourself?

WIL: I've read through the docs but I haven't played with it at all. It looks really nice.

TORAN: I think the joke that was off the air last time I was on, Wil you might remember this. You're on that podcast with Charles. I said something like, "I'm not young enough to actually be working with Glimmer," and I felt that way for a long time because one thing you should know is it's a pre-1.0 and if you guys have ever worked in a pre-1.0 ecosystem, myself the biggest experience I have to draw from is really pre-1.0 Ember and there were some big changes before 1.0.

You can imagine back to that throwaway comment about being very young, there was actually a big change in Glimmer itself recently where they decided to... I don't know if the right word is Pascal Case but they've literally gone away from that 'dasharized' components. It used to have 'foo-bar' and in your template, you would actually see lower case of 'foo-bar' and now that would just be all uppercase. Well, not all uppercase but 'FooBar' and no dash, which is a big change recently.

WIL: So a class case, kind of like React or JSX.

TORAN: Yeah, exactly. They have a great blog post. Actually, we can reference that in the show notes, about some of these big changes in that release. It was Glimmer 0.8 so it's still, it's making its way to the 1.0 but I got interested in this really for two reasons in the last couple of months. The first was, if you actually go build something with Glimmer -- and this is my experience -- is for the novice programmer just taking a look at it, it's really just a way to use web components to build an application. There's no routing. There's no opinions, really. There's no services like you have in Ember or contexts like you have in React. The first challenge you run up against is when you get beyond a single component or two components and suddenly, you need to share some state across this application. How do you do that?

If you guys have some experience, I know the Frontside, with React, if you're not using MobX or Redux or something like that, a lot of times you'll see this pattern where you're actually passing a piece of state through the entire tree or the shared state through a big part of the component tree. Of course, that becomes painful as the application gets to a certain size. One of the things I thought about is if I was to build a real application, the one in mind that was certainly not built yet because I'm not using Glimmer at work but I always think about the ember-rideshare.

I know you guys had Alex Matchneer, recently talking about routing and Alex mentions in that episode this challenge for the Ember router today, to be reactive to server-sent events. In the case, imagine you have an Uber or a Lyft app and after the ride is over, the server wants to send an event, maybe and then the app needs to react to that event -- sending you maybe to a new route or sending you back to the map to pick a new ride. The gist of the ride share app is, and Alex, of course I would reference anyone to that podcast, he does a lot better job describing the routing challenges and those are a little bit out of scope for this discussion, but imagine you're going to build an app that ambitious and you're going to build a Lyft in Glimmer.

What I found was missing is really what we take for granted in Ember, which is the Ember Service and that is like a singleton or an object that allows you to have a piece of state and then share that state around by injecting that service in only the components that need to reach up and grab that shared state. Redux, which I know your audience is pretty familiar with but a quick recap, Redux is just kind of a global JavaScript object that has state and there's often a library that lets you connect to that state so you can use it in your various components. Glimmer Redux is no different than that, actually. It just allows you, instead of having to create maybe one global JavaScript object and kind of pass it down the hierarchy, you can instead just connect the components that need to be aware of Redux. The ones that don't, of course they just don't connect.

ELRICK: I know that you had a hand or build ember-redux and now you build Glimmer Redux. Were there any challenges between building those two different add-ons into two different ecosystems?

TORAN: I should take one minor step back because that question is a great segue. I didn't want to touch on the second motivation for Glimmer Redux, which is actually really closely related here and that is, of course I did write ember-redux so with every open source project, there's a little selfish motivation here. I imagine last year when Glimmer was announced at EmberConf that there would be a story from Glimmer to Ember. The idea being you’re a small startup, you just want to get your web application going, you don't necessarily like all the big conventions or you just don't think you need all of Ember when you get started but six months down the road, you're suddenly looking at tree shaking and lazy loading with engines and you're thinking, "I wish we had that."

Realistically speaking, like today what would a transition for a Glimmer shop to Ember be. Honestly, I think it's tough without a library like Glimmer Redux. Of course, I wrote this with pretty much a mere of the connect API. If people were to check out the ReadMe of Glimmer Redux and ember-redux and you looked at a connected or redux-aware component in both of those cases, the best case scenario is the only difference would be in the import. Instead of import connect from a Glimmer Redux, you would be import connect from ember-redux. Everything else underneath, all the ecosystem of Redux that you can use and both are completely compatible. In fact, if you were to move, imagine you Ember new and you're thinking, "I got to move my Glimmer ride-share to ember-rideshare.

Since Redux does a good job in encapsulating all of the state transformations in vanilla JavaScript, you don't have to really worry about the differences between a number object and not having Ember object. After you did Ember new, essentially you would copy over your components. For the most part, they're still template-driven. A lot of handlebars and a lot of TypeScript to JavaScript is the biggest mismatch you would have between Glimmer and coming over to Ember, of course but there is Ember CLI TypeScript or Ember TypeScript, I think. Long story short, you essentially copy the directories of your reducers or middleware from your Glimmer app over to Ember and there should be no changes necessary at all.

ELRICK: I know that is the dream that you touched on, I think they phrase it as 'NPM installing your way up to Ember.' In your perspective, do you think that is going to be a thing? Is it going to get there? What are your feelings on that?

TORAN: I would definitely be in trouble if I didn't say upfront that I'm not on the Ember core team. Sometimes, people get that confused for some reason but I don't speak for the core team and I'm not really privy to anything. But I do think that the core team has this in mind that there will be a set of NPM installable modules that eventually land you the full set of tools and abstractions that we see in Ember today. A big one that I'd hit on earlier is services. What will services or the Glimmer 0.5 version of services look like when it lands and what about routing and those sort of things?

This was really my personal take on how could I make that migration right now without asking permission from the core team. In a Glimmer Redux, I think honestly it offers a good 80% of that. You still have the routing issue, which is a little challenging and you have the TypeScript issue, which you just have to be aware of some of the limitations of using TypeScript in Ember.

ELRICK: That's an excellent point that you made because when people outside of the core team take it upon themselves to then try to implement different things around the ecosystem, that can then be motivation or an example for people outside of the core team to see like, "This is a possible solution to a goal we're trying to reach." Kudos to you.

TORAN: Back to your original question about 15 minutes ago, what was different about the ecosystems writing ember-redux the add-on and Glimmer Redux, which is... I don't really even want to call it an add-on. I kind of label it as a Glimmer library but to your point just a second ago, when I got interested and saying, "I wrote Glimmer Redux, now I want to share it so other Glimmer authors don't need to copy/paste this file," and the first resistance I hit up on is there really is no Glimmer library or Glimmer add-on. You could write an Ember add-on because if you guys get into Glimmer, you'll find this as well. There are certain hooks that are used in the Glimmer build process where Ember add-ons like Ember CLI SASS can be used from Glimmer.

But the challenge I had using an Ember add-on, of course is that this wasn't an Ember component or anything Ember related. It needed to really be test driven from a Glimmer apps. Really, if you went to NPM installers, what you're pulling in is effectively my Glimmer app that also exports publicly this connect function, which is not necessarily you're leading, maybe anything to core team that could happen. A big reason for that as well and one of the challenges building this, is that Glimmer right now, after to try to emulate my 'quasi-success' in doing this, is really bring your own build system, to actually share a Glimmer components or internals like this connect API that Glimmer components can use.

What I mean by that is on the website, one of the challenges I faced was -- this isn't a knock against the core team, this is just my honest experiences -- when I read the Glimmer JS docs and it says right in the installing guide, "Glimmer uses Ember CLI, this battle-tested command line interface from the Ember project." Now, pause right there because again, not to beat up on the Ember core team or anything but assuming in that one sentence that they're using Ember app, what I noticed when I opened the Ember CLI build file is the project was actually a Glimmer app. Now, I did a little bit of digging here and I think this is validated, at least back when I worked on Glimmer Redux last month, that Glimmer app is not like inheriting or pulling in a bunch of shared code from Ember app. It's actually a completely separate build tool.

As part of this process, I actually for the first time had to go through and learn Rollup and understand how the Broccoli process is kicked off, how both Rollup and Babel are used to build this and then how to apply some convention. If you guys are familiar with Ember, you have this add-on directory and an app directory. The guidance from the Ember team is around how to structure add-ons. Of course, you write all of your kind of private-ish or add-on code in the add-on directory and then whatever you think will be public, you export from the app directory and that sort of merges it into the tree when the application is built. People are allowed to use your code but then also, they can override that code.

One of the challenges here is if I wanted that exact same API and I want people to make this migration from Glimmer to Ember using Redux, I had to actually invent that convention so I wrote a Rollup plugin and it's all listed in the docs here. One of the strange things people who are checking out Glimmer Redux hit me with first is, "I see a Yarn installed Glimmer Redux but then second step is installing this Rollup plugin. What's the deal?" I think it is because most people assume the Ember CLI you're using is identical and somehow, I should have written an Ember CLI add-on for this. I think that was the biggest learning curve and people should just be aware of that.

If you're interested in sharing code right now, there is really not a baked story and that's okay. It allows people to innovate. Of course, the innovator's dilemma being that, I don't really know without [inaudible] some migrating RFC or getting involved with the core team, how to make that thing. I ultimately just hope the core team improves this and I'm sure they will but for right now, I don't really want to wait around for it.

ELRICK: Got you. What is Rollup, for people that are not familiar with what Rollup is? I'm sure everyone is probably familiar with Babel by now because it's used everywhere but what is Rollup?

TORAN: Embarrassingly, I don't know the technical... But I would say the role that you can see, if you actually step through the node process as your Glimmer app is being built, is it helps really condense the overall build size. What I see is it's essentially traversing like Browserify in some ways. This is, again just my primitive look but it traverses all the imports you have and it tries to pull in the bare minimum to keep the bundle size as small as possible.

ELRICK: Toran, you have had a lot of experience with Redux and Redux is now being used in several different software platforms, I guess or software areas like Vue and they probably even have in Angular now. They have it in Ember and React. Redux is kind of spreading its wings and it seeds across every ecosystem. Do you feel that Redux has reached a state where people are just satisfied with the current state of Redux or do you think that people are going to be then, looking to build another abstraction on top of Redux? Do you have any thoughts on that?

TORAN: The fact that Redux is so simple has allowed it to become so ubiquitous. I heard someone say this term the other today, which is like, 'ubiquity over consistency,' and that I think describes the both the growth of Redux and why it is kind of de facto for data management across all ecosystems. I think there's two camps that I hear about and I'm curious if you guys see this in your consulting work but there is certainly, the developer see this ubiquity but no consistency and see chaos in their experiences. I can totally relate to this.

There are development shops I've seen where one team goes this direction because there's no strict guidance goes another and then when those teams meet up for a project in 12 months, they look at each other and the apps are, of course nothing alike, which is a big problem that Ember tries to solve. My biggest question here really is kind of curving us slightly back to the Glimmer story.

If I can reframe your question, Ember is traditionally very big on convention and I think a lot of the community that is still in Ember today in large part is because of this convention, these guide rails about the community has set up but Glimmer being this NPM install your way to Ember, I think along the way, there's going to be either a new set of users that are coming just for the winds of the Glimmer VM and they happen to find themselves, not necessarily in love with some restrictions or opinions that would come with a migration directly to Ember and I'm curious if the Glimmer community that will show up for that is mostly Ember backed, meaning that they want to slowly build with RFC as a process that the entire community uses and there's one solution like we end up with often an Ember. If a community evolves more experimentally like you saw in React, where there was, of course Redux but then there was MobX and then there was various little wares for Redux that different teams would try out and eventually, there was no one proven way but there was always at the heart of it, Redux with some extensions or other add-ons around it that got teams to where they are today. I'm curious to see where this Redux, especially with the Glimmer influence, will end up.

WIL: You touched on a little, I think one of the, maybe not a problem necessarily but one of the biggest barriers in Redux and React and maybe Glimmer -- I'm not sure -- is that it's almost too loose without any opinions at all so it kind of gives developers the freedom to mess things up big time. What's your opinion on that with Glimmer, like Glimmer is headed toward this NPM install? Is it too loose?

TORAN: I guess that's the question, I wish we both had the answer to. It's funny. It's a double-edged sword. If it's loose, it seems like somebody is going to go create a mess but at the same time, if it's loose on the surface, it often seems like it has less surface area and as a result, a lower learning curve. React is a good example where most people, I think have gone to React or at least in my experience, I like React because it was so simple and there wasn't a huge amount of things to learn. There wasn't this full ecosystem, at least out of the gate but of course, what you find the moment you want to join a team or go build something ambitious is that you've got to make a bunch of decisions and that is certainly the calling card of Ember. What makes Ember special is they've settled on a handful of those decisions.

I think ideally, I'd like to see the community in Glimmer check out Redux, get an idea of what problem it actually solves for them and if it is useful, then find a set of middleware or extensions from the wider ecosystem that actually solve the problems they face. Back to this Glimmer rideshare example, I think one thing that stands in the way as I play around with that is just a very basic routing story. Even if that is as simple as I have a component that I'll just call it route, what hooks in this Glimmer component are correct to fetch data.

In the Ember ecosystem, we have a very special route object, essentially who has a set of hooks that are known for handling the asynchronous stuff in our Ember apps but there isn't anything like that yet, in Glimmer which is the next stumbling block for me to actually go build something big. I'm kind of messing around with the idea of this reactive router built out of Glimmer components but until that's actually kind of surface, mostly just spit balling here.

ELRICK: With flexibility comes a lot of power but then there's also the case where our flexibility could lead to chaos. But being that something as flexible, it allows you to adapt it to whatever your particular needs are -- you know, the 'special snowflake' as everyone ends up being. Because Ember has been around for a while now and has proven itself and its battle-tested in a lot of areas, now as NPM install in our way from Glimmer up to Ember, do you think that they'll be able to then, extract some of those hardened pieces of Ember out of Ember and then give us a solution inside of the Glimmer ecosystem based off of the Ember ecosystem? Then not have so many varying different opinions or different packages or add-ons or libraries that you may need to pull from that you may just have like a set of Glimmer-approved libraries or add-ons to use to then, get your way up to this full Ember app. Do you think that that's a road they're taking? Or a possible road?

TORAN: Yeah, I think for sure. A lot of the talk right now if you're in the Glimmer channel on Slack in the Ember community, I think the next big step here is having the ability to take a Glimmer component as it exists today and use that, extrapolate from there and the plan is to prove out things in a more experimental ecosystem as pre-1.0 Glimmer ecosystem. As they become more solid, we essentially just adopt them and then use them as a first class. In Ember actually, this isn't true but I envision a world where in the months ahead, we're essentially using Glimmer components in Ember so I'm already challenging myself with how would a library or add-on author help people make this progression from Glimmer to Ember if they don't do something like I've done here with Glimmer Redux because eventually, there will be this shared component world, at least in the middle ground, where Ember has both Ember components and Glimmer components.

I think it’s a road yet to be traveled and that's what I'm excited to be on the podcast and get people talking about building libraries for Glimmer, just because there is not a cow path or a paved road for you right now. I think if you learn just a little bit of Rollup or you dive in and take a look at the DI system built in a Glimmer right now, there is a lot you can actually do with it. I know there are certainly big named add-on authors already checking it out in preparation for such a migration.

WIL: Besides the whole Rollup process, was there any other friction points in integrating Redux with Glimmer?

TORAN: One that I want to call out but it is, I believe still Rollup related. It's mostly a PSA, to warn people. If you are building one of these little libraries like I talked about with Glimmer and you're going to do a Yarn link, which means that you're going to just work locally and not really publish to NPM until you're done, be aware that if you're Yarn linking and then going over to another project to test this Glimmer library, then you'll actually get temporarily two copies of Glimmer. In fact, that is how this Rollup plugin I wrote -- sort of 'bring your own build' for Glimmer -- became essential as I was like, "I'm getting two copies of the Glimmer component," so what was happening is example, I was playing with this connect API, it was always calling into the wrong Glimmer code so the code that I was expecting in my add-on was never firing. We come to find out when you're not doing Yarn link or you're not working locally, this isn't a problem.

Actually, Tom Dale reached out and helped me a little bit later because my first version of the Rollup plugin had this little hack in there that said essentially, "I'm going to mask away this duplicate Glimmer problem," and it turns out it's not a problem. If you are working locally, be aware that you will have this problem as my guess.

ELRICK: What gets you most excited using Glimmer? Are there any specific features or things within Glimmer to get you most excited? I guess the second question would be, what do you think Glimmer is going to unlock for the future of app development in Ember?

TORAN: I think the first thing I get excited about was visible in this talk that Tom Dale gave a couple weeks ago. I'll try and dig that up for the show notes where he show the advantages of Glimmer over the competition today. The big thing that just blew me away was some of the advantages over, even Preact, which I was kind of surprised that a lot of times there's this rivalry for performance especially between React and Angular and Ember but no one ever really talks about Preact, which is known in a lot of ways as the thinner, lighter-weight React, if I'm not completely wrong. I'm sure somebody is going to be table flip on that definition.

But my view was that if you were really performance-hungry, you could check out Preact, which was for performance reasons there was a smaller bundle size so you're just actually shipping less JavaScript, which means they're parsing less JavaScript and Tom went directly after this library in his talk and just showed a very interesting point at the end. I don't want to spoil it for people but let's just say, it appears to show Glimmer as just an order of magnitude better as a primitive for building web components. I think that is the big draw and how will that make Ember better. I think, mostly in the same way that I just described, where we'll essentially get a component for free in Ember that is just a better performing primitive for the web.

ELRICK: I'm not too familiar with the guts of Glimmer but from what I understand, Glimmer compiles down and it has some opcodes that then compile down to binary. Is that correct?

TORAN: Yeah, I think there is a binary format that they're shipping or they will soon be shipping. If you're really interested in the technical details of that, I'll definitely be sure you check out this talk from Tom because now the real magic behind this is they essentially boil it down to the ability to compile these templates down to 'byte code,' as they call it and Tom has a really funny part in his talk where he says, "You know, it's not a marketing term, this byte code word," and it becomes true because later in the talk, you hear that, essentially the Glimmer VM or the big value add of Glimmers VM is that you're just feeding it with byte code until the next 16 millisecond buffer comes up to paint, in which case you just pause and that allows, I think as he describes, less jank or allows less freezing up that main thread because you're releasing control back to the UI every 16 milliseconds, essentially.

ELRICK: Yesterday, there was a meet up with a lot of the Ember core team. Ed Faulkner had made a point that since Glimmer compiled down to byte code that then is not too far of a stretch to then use that with something like Web Assembly, that with then give Glimmer an extreme performance boost. He sets up Glimmer to be used in what can be potentially the future of the web, which is Web Assembly, which I thought was really interesting and it kind of blew my mind to think, "Oh, yeah. That is true since it compiled down to byte code and Web Assembly compiles down to that." We can get that performance boost in that Glimmer is forward thinking in a sense.

TORAN: Yeah, man. I totally agree. One of the things I honestly value about the Ember core team is they're very both tactical and visionary at the same time, where in the short term, they're doing things to accomplish real pain points we have but without anyone really realizing it, they're also setting up the stage to solve huge problems that we're all going to face in six, 12, 18 months. I definitely appreciate and love the work these guys are doing. They should hear about it. Obviously, this is all open source and most of this time is gifted, just given away so I really appreciate the core team.

WIL: Speaking of performance, we know that the Ember has a run loop and basically, most of these libraries have some sort of loop that determines when they should batch things together or render things to the screen. Does the Glimmer have this concept? What do they take advantage of to make it as performant as you say?

TORAN: Yeah, that's actually a really good question and I'm probably minimally equipped to answer it but the short version of it is there is no equivalent run loop for batching work that I've seen inside of Glimmer. Instead, you're more directly interacting with request animation frame, which we miss directly from Mozilla here but requests animation frame tells the browser you [inaudible] browser call specific function to update an animation before the next repaint. Where does this come in for Glimmer?

Essentially when you call set or you decide to change a property that Glimmer is listening to and if anyone was not familiar with Glimmer, you designate this by using the tracked attribute. The tracked attribute, when you change a value, it fires this 'set property did change' and behind the scenes, 'set property did change,' if you are familiar with Ember, in my mind is close to 'notify property change,' which is what happens when you do Ember.set.

If you've ever actually change something in Ember behind the scenes, there is a notify property change event fired and then we queue that work and it's a similar-ish process except that there is no run loop. What we do is we just call schedule rerender, I think in Glimmer and that just fires off request animation frame to try and rerender within that 16 millisecond window before the next repaint.

WIL: From my understanding, the request animation frame is Glimmer's run loop essentially.

TORAN: I think I saw actually a discussion between someone at Ember core kind of saying in the public channel that, if you're using Embers run loop, the equivalent-ish today would be request animation frame but the point came up that there's really no way to have a different set of cues because Ember itself has many different cues in the run loop and request animation frame, as far as I know really is just one function with a single callback, where you try and fit in as much work before that repaint as you can. I don't think there's prioritization that you would get in the Ember run loop. But at the same time, I'm not sure if that's actually a requirement. I don't think request animation frame was as mature as it is today back when backburner, which is behind the scenes of what Ember run loop is using, was built years ago.

ELRICK: Since Glimmer is using requests animation frame and not the Ember run loop, is it going to continue to use requests animation frame in the future or they're going to develop like a run loop equivalency for Glimmer?

TORAN: That's definitely out of my depth. I know from looking inside the code, the rerender work essentially calls like a 'begin,' to say begin doing your work, which I believe is like the reconciliation type work if you have a React background, I believe. I don't know what decides to end that. If the request animation frame is truly saying, "We're at the 16 millisecond budget and we're going to quit feeding the Glimmer VM byte code instructions now because we need to go back and paint." I would guess that that is the high level narrative but I actually don't know the implementation details.

ELRICK: Since Glimmer is pre-1.0, it's a place for you to experiment, try out new things, really exciting area to play around in. What kind applications do you see people building today using Glimmer and what's the good application that's a good fit for Glimmer right now for you to experiment with?

TORAN: The big application that I've seen that exists is being built with Glimmer in its current form is the Glimmer Playground. The Glimmer Playground is an area to go mess around with Glimmer, if you've never used it or you just want to go [inaudible] with the basics. As far as what is an ideal application, honestly I think we're at a point where we need to push the bounds of what a purely component based library can do. I think if you could come up with some kind of basic routing story and have a mechanism to share state and bubble events, whether that's Redux or some kind of home grown service layer at some point. That would allow you, in my opinion to build just about anything.

The only caveat that you're going to be missing is a ton of opinions and you're going to be paving new grounds so just be aware that happy path for any pre-1.0 is be aware that they could change anything, anytime. But that shouldn't really restrict you from making a hobby project out of it.

Back to your original question, I would say building any app that you're okay to go back and rework, not necessarily reinventing Facebook or something like that with it at this moment but at the same time, it would be great if someone did in a hobby way because we need to see some of the constraints and challenges. I got playing around with it myself and noticing if I'm going to build anything with Glimmer, I've got to have a way to share state and bubble events up to the single atom at the top that allows me to share all state. I think without some experimentation, we just won't know what apps are possible.

ELRICK: You've been talking a lot about Glimmer today and using Glimmer. Since Glimmer is pre-1.0, are there any limitations that people should be aware of when they're going to be going into building a new Glimmer app?

TORAN: For Glimmer Redux specifically, one of the challenges that people would see and they should know about if they're going to use this little Glimmer library, is that it doesn't yet allow you to write reducers in TypeScript, which would be a little counterintuitive because if you get into Glimmer, you'll notice the big differences -- everything is in TypeScript and not JavaScript. Luckily though, I think this is really just a build tool decision. Truthfully, the spike version of this that I have where you can use TypeScript, it doesn't require any change to the Glimmer Redux library at all. In fact it's completely unchanged. The difference here is that the Rollup plugin I use needs to see a change.

It's kind of weird in that way, back to my point earlier where you kind of bring your own build chain and one of the things I don't like right now, which is the reason I haven't published this TypeScript version of it is that I'm actually doing a TypeScript compile of all the reducers or middleware in the Rollup plugin. With the mass confusion right now between Broccoli, Rollup and Babel, I just don't feel really great about it. Mostly because I have not truly been in the guts of the Glimmer app build tool or the application pipeline yet and I want to be a bit more educated there before ship something, back to being a Midwest developer here. I just don't feel good about shipping something and say, "Oops, we got it wrong."

I take a lot of time and I also take a lot of pride in what I am shipping so I want to have a really good story about TypeScript. It does make for a little bit of a weird experience: bouncing between Glimmer components written TypeScript and flipping back to reducer file written in JavaScript. just be aware of it and at the same time, I didn't want to completely halt shipping it because again, I think we need to actually build apps with this and a concession here being that, of course you have to write reducers in JavaScript was still enough for me personally to get some value out of building Glimmer apps and honestly, it got me building them sooner.

WIL: Is there a way to use Glimmer Redux without the Rollup? Can we import something into our Glimmer app to use it or this Rollup is required?

TORAN: Yeah, that's a great point. You can omit the Rollup plugin entirely. What that results in is you'll have to do some hand wiring yourself. One of the upsides or one of the benefits of this Rollup plugin that I wrote is this conventionally provides a store and redux-thunk, kind of like a happy path for people who are just not familiar with wiring up their own Redux store. If you forego this plugin, you just have to do that yourself. You may actually have to do some kind of Rollup hacking in your Glimmer app, which is the thing I want to avoid.

The one in particular that I know you'd have to do is there is a Node ENV that is looking for a production setting in Redux so the first thing you have to do is use a Rollup replace plugin to replace Node ENV with Ember ENV. If you can't do that, you actually get an error in just trying to stand up your Glimmer app with Redux.

ELRICK: Toran, are you giving any talks or have any books or anything that you want to get out there and talk about?

TORAN: I'm not actually on speaking circuit right now. I am certainly, probably like you guys are, thrown a talk or two together for the EmberConf proposals that are now out. I think they're open until November 21st. If anyone is thinking about submitting a talk to EmberConf, this should be in next March. Now is the time to get those in and I certainly have one out there but I've got one, off the top of my head that I would certainly like to find some time and submit that's related to Glimmer.

ELRICK: Cool. We had a wonderful podcast today. We touched on Glimmer Redux and Glimmer and I want to thank Toran for coming on. Thank you, Toran.

TORAN: Thanks for having me, guys. The fifth time I have to be on, I don't know if that will be in 2017, though.

ELRICK: Yeah, we're going to bring you back for a fifth time and I would also like to thank Wil for coming on the podcast as well. Wil, thank you.

WIL: Thanks for having us, Elrick.

ELRICK: Anytime. Toran, if people want to reach you, is there a particular place on Twitter or anything that people can reach out to you or email or anything?

TORAN: Yeah, at GitHub, I got my email out there but also on Twitter, of course. You can reply me there. If you have a question specifically about Glimmer Redux, of course you can got to GitHub and throw an issue up there or hit me in the Redux channel on the Ember Community Slack.

ELRICK: Thank you all once again for listening. This is the Frontside signing off and if you want to reach out, you can always hit us up at the Frontside.io and we always want to hear about your new project that you're working on. Thank you for listening and that's peace from the Frontside.

WIL: Everybody have a good Thanksgiving!

Kaylie Kwon: kaylieEB | [email protected]

Show Notes:

Resources:

Transcript:

CHARLES: Hello everybody and welcome to The Frontside Podcast. My name is Charles Lowell, I'm a software developer here at the Frontside and your podcast host-in-training. With me today is Elrick Ryan. Hello Elrick.

ELRICK: Hey, what's up Charles?

CHARLES: Not much. Are you enjoying your morning so far?

ELRICK: Yeah, my morning is going well. Everything is good.

CHARLES: Lots of cups of coffee?

ELRICK: No cups of coffee yet. I've been drinking a lot of green tea.

CHARLES: I've actually heard that's really good for you.

ELRICK: That's what I heard too. That's why I started drinking it.

CHARLES: Did you continue because it tastes good or do you just live on the idea of how good it is for you?

ELRICK: A little bit of both. It doesn't taste that great but it's not horrible. It's almost like an acquired taste and then when you add in, "This is good for me," then it tastes great.

CHARLES: Okay. We got a nice [inaudible] there.

ELRICK: Yes.

CHARLES: Anyway, I guess we should, at some point get on to the main content of our podcast. We have a very interesting guest with us today who has her fingers in all kinds of pies that we were talking about just at the pre-show, just before we were recording so we're really, really happy to have Kaylie Kwon. Thank you for coming on the show and welcome.

KAYLIE: Hi. Thank you for having me.

CHARLES: It's going to be great. Now, you are a software engineer at Eventbrite, that's correct?

KAYLIE: Yep.

CHARLES: What kind of things do you do over there?

KAYLIE: I used to work on part of the feature team that worked on their reserved seating product but not too long ago, I moved to our frontend platform team, which is a team that helps other frontend engineers move faster through things like working on infrastructure or Eventbrite design system and dev tools.

CHARLES: So like getting into the tools that unlock the exponential productivity of the developer team?

KAYLIE: Uhm-mm.

CHARLES: We're going to dig into all of that because you just listed a bunch of really interesting stuff. I'm really excited to talk about the design systems, in particular but lots of different stuff. But before we do, I understand that you have a fairly unique way of entering in to the position that you're in now. Your journey didn't follow the traditional arc. Would you be willing to elaborate on that or tell us that story?

KAYLIE: Yeah, totally. I graduated with a degree in art history, not related to computer science at all. Then right after, I moved to New York and worked for a small startup. It was marketing/business development role and I wasn't really happy with it. I was working on some design-y stuff on the side with HTML and CSS but I just felt like my brain just needed to be stimulated a little more. I applied to this all women's bootcamp program called Hackbright in San Francisco. It was a three-month intensive program and luckily coming out of that, I had at least some initial knowledge to get my foot in the door and Eventbrite was one of the hiring partners.

They brought me in for interview. I actually had no idea that it was going to be a frontend engineering role because my bootcamp was totally in Python and it just had more with a backend focus. Ben, who spoke at React Boston Conference with me, was actually one of the interviewers and he gave me this Clojure problem and I solved it but in Python just using recursion. He was like, "You got it. Just convert it to JavaScript," and I was like, "No, it just can't be done. No JavaScript."

But there's the reason he would chose to hire me and they onboarded me as an internal bootcamp within Eventbrite. The first three months I was there, I learned about React, Redux, JavaScript, ES6, all of that good stuff. Then they moved me to a feature team, where I continued, I guess working on the product and then I became involved more with open source projects. I really expressed interest in when you're a new engineer and coming onboard, there's all these assumed knowledge that isn't documented anywhere or something will be really obscure and hard to use but people will just assume that that's the status quo.

This idea around developer experience and helping other developers move faster, it just kind of become a natural interest of mine. I was talking to the platform team, Ben in particular, expressed my interest in working in these areas. Just about like a month ago, we did a big re-org and I landed on the platform team. Currently, we have a lot of projects in flight. One of them being moving our dependency system from our old codebase, which first was written in RequireJS to webpack. We're building out our Eventbrite design system, which is basically a shared UI component library that other teams could leverage.

Our platform team will just come in and make sure that the API is usable across different teams and maintain a consistent brand in terms of look and feel. We're also working on other tooling stuff like making sure we use Docker for our dev environment and making sure that the frontend containers don't break, making sure that everyone is on the same version of ESLint, Node and things like that.

CHARLES: How do you make sure that the frontend containers don't break?

KAYLIE: It's actually hard. I think one thing that we're trying to test right now is using Yarn Offline Mirror and having better caching. When you build a container, it'll look into the cache directory, which is just bunch of committed tarballs. That way, they don't have to fetch to the network each time because once the lockfile or package.JSON changes at our current state, it'll rebuild the entire container and it could take a very long time. We just have a lot of packages that we've added over time. Other things, we're experimenting along with our platform team on the backend side about having remote images. Instead of devs building their containers locally on their computer, having them remotely built by a CI system and then just pulling the container and the images down.

CHARLES: Really, really you're like all over the place. That sounds so much fun.

KAYLIE: Yeah, it's a lot of context switching. Sorry, if I was jumping too many topics at once.

CHARLES: No, absolutely not. I think it's actually fascinating and it's actually capture the kind of scope because when you are doing development, if you yourself are not running up and down the stack, the tools that you use are. The better the tools, the better you're able to focus on the one little sliver of the stack that you're working on. If I don't have to worry about where are my Docker images are coming from or if there's a temporary network flip that I can't install my dependencies, if I don't have to worry about those things, then that makes me more effective so it's important to just kind of lay out all of the stuff that goes into making a quality developer experience.

KAYLIE: Yeah, the dream is like frontend people wouldn't have to worry about any of the backend stuff and they just have this isolated environment, where they could just work on what they do best, which is JavaScript and writing features in React. Everything else just works and they could see that replicated through their dev environment as well as QA and Prod and hopefully, make every tooling that they use, like testing and linting all easily, intuitive and accessible.

CHARLES: How is it that you're working up and down the stack, making sure that your CI systems, your Docker images but then also at the same time, working on a design system, which you've got collaboration, I assume with some pretty hard core devops teams but also then you're interfacing with designers. You kind of flesh out your design system. Are those the same people on the platform team? Or there are different groups within it?

KAYLIE: We have designers and researchers actually, as part of our engineering unit. We work pretty closely with them to define guidelines on what the design system should look like because coming into making this designs system, one thing we really wanted to make sure that is that both sides of engineering and design have input, rather than the previous old version called Style Guide, which was more engineering-driven that not. One team would need a model so they would build a model and a different team would build a slightly different model and we would end up with five different models. They want to be maintained over time and there wasn't really a focus on accessibility or consistency of brand so the design system project was to eliminate all of those pain points.

CHARLES: For people who may not be familiar with design system but it's something that certainly is cramping up more and more, what's the general strategy you take? Clearly, you talked about the kind of the pains that it solves, like I'm experiencing this pain where I've got five dialogues, I've got three ways of laying out forms, I've got these problems. What's the strategy that I go about as my organization for trying to implement this? Like now, I want to do a design system, where do I start?

KAYLIE: One big thing, I think that helped us was developing Eventbrite design system as just the standalone product. You could run Eventbrite design system as a standalone with all of the documentation and components with each of their different props that could be configured and it has its own set of tests. All of the components are API-driven so there's nothing specific about business logic that it assumes. For example, our button component just assumes that I'll be receiving a type of submission and some [inaudible] body. It doesn't make any assumptions that it's going to be anything Eventbrite specific. It's high-level configurable that the end user wants it to be.

Another thing that helped is we have a planned approach session right before we start working on a new component. A developer who would be building the component would meet with one of the frontend platform members and will be discussing the API of the component, the CSS that would go in it and the designer would do the final QA. The development takes longer than if you were to write just a component for your app but we're trying to build it out for a long term use. We have versioning for Eventbrite design system as a library so whenever you make updates, we added to the changelog and then it gets released and we bumped in Core, which is where all of our apps live and people have to get the new version. Basically, it's an orchestrated effort and we have a process built around scheduled releases and bumping it in Core.

CHARLES: I get the wanting to have a uniform buttons, uniform labels, dialogs and things like that. How do you attack the problem of higher-level workflows like a form submission and validation process that might have a lot of different pathways? How do you guide that using a design system?

KAYLIE: Those are actually really good questions because we went through issues like that. Validation form field or components that have logic around more complex inputs and validation is really hard because different teams use it in a slightly different way so we ended up having two versions of complex inputs. Now, we're in the process of deprecating the V1 and then having people move over to V2. We also use the Atomic Design Principles for Eventbrite design system. It means, things like atoms would be composed of buttons and we have molecules, which are slightly more complex. Then one level above that would be organisms, which are things like forms.

Obviously, as you move up to like the higher level, it becomes more difficult for us to decide does this belong in EDS or does it belong with the app. There's lots of refactoring that sometimes ends up happening as a result of something built one way and us realizing later down the road that it wasn't actually very extensible and we just built it for this particular use case. Sometimes, it's the opposite where this was too generic. We can make it a little bit more specific and add more logic to it. It really depends on the component but that's been our strategy, just taking the components as needed. We're still in the earlier stages. We've released EDS a year ago and now, we're almost close to finishing it. We're just missing a handful of components. They'll be more [inaudible] that's available as opposed to adding a brand new component.

CHARLES: Did you find guidance in existing design systems? One thing is why not just use one of the existing ones that's out there. Is it a matter of branding? Is it a matter of, "There are certain interactions which are unique to our product that we need to design system to capture them?" I guess my question is, if this is something I want to take on myself, what tools would I be able to reach for and what other design systems would I be able to look to versus what I have to contribute myself? What am I going to be looking for to share with the community and what am I going to look for that's develop that's uniquely mine?

KAYLIE: I think consistency was a big issue. Sort of the genesis of idea came actually even before I joined the company but I think what they were going for was that we already had an existing component library called Style Guide and it wasn't really working out for us. To build like a common UI library was a natural assumption but making it better, making it more reusable. We looked at companies like Airbnb and Microsoft and I learned a lot from what they were doing. You definitely wanted some components that are specific to Eventbrite such as a ticket card or a media card that we use for our browse pages which would be needed for Eventbrite specific pages.

I think mostly, the designers wanted more control because relying on third library means we don't always get what we want. We're actually thinking about making EDS open source at some point in the future, where it could take themes so other companies or individuals can leverage it but use their own theming scheme. If Spotify came and wanted to use EDS but using their colors and brand logo, then they could do so, just by layering a different configuration style to it.

CHARLES: That's such an interesting idea. Is this something that you all have explored, maybe collaborating with another company? I'm trying to think what would be the benefit of making it publicly available, unless you would be getting lots of contributions back to improve EDS itself. Is that the idea?

KAYLIE: Yeah. We're definitely set on making it public at some point in the future but making it open source is a different conversation because like you said, there's pros and cons that comes with open sourcing a package. We recently open source britecharts, which is a D3 library. It's been getting a lot of contributions. It's a good recruiting tool but it's also a lot of maintenance work outside of developers normal work hours. Also, we have to start worrying about like when we make a breaking change to a component, we're not just breaking it for our own product but for other developers.

We currently have external developers. We have Eventbrite plugin system called Spectrum so developers are already building on top of that and they've been asking for something like this, where they could leverage and match the look and feel with the rest of the product. The downside is lots of maintenance hours and worrying about all the people that would be breaking the component for by just solving your use case.

I feel like I didn't fully answer one of your questions, which was you have a different dynamic of people working on both really backend ops things, as well as a really frontend design system work. We definitely got very smart people on the team. Some people definitely have expertise in one area versus on others. Currently, we have five developers and some of us lean more towards the backend of the frontend work and I am one of those people working on things like Yarn and Docker and build system work but it's funny because I was thinking if I had a portfolio then, there wouldn't be any visual components to it, even though I'm a frontend developer. It would just be like terminal screens. We try to divide the work but everyone tries to, I guess develop, at least some shared knowledge around why we make the decisions that we do.

CHARLES: It's interesting how they all intersect. I feel like the trend of the last 10 years has been to dev of all the things. I think the first thing was this artificial divide between developers and testing and that came together with the test driven development and test obsessed. Then there was this divide between developer and an ops and that divide went away now with the advent of the devops movement. Now, there's this divide between developers and design and I feel like that kind of wall is collapsing right now. You have developers participating a lot more in design and designers spending a lot more in development. We're seeing that but it's just funny how the devs starts integrating with all the things.

KAYLIE: Yeah, that's a really good insight.

CHARLES: You said you kind of naturally gravitate towards more of the backend doing the working with Docker, working with Yarn. How did you get actually involved with the Yarn Project?

KAYLIE: Eventbrite converted over from NPM to Yarn maybe a year ago and the benefits that we got from converting over was awesome because we were manually editing NPM shrinkwrap, which is a nightmare and the installation speed of the container was really slow just because we were on NPM and it didn't really have any advance caching mechanisms at that point. Yarn just sped up a lot of things for us. I really like the user interface outside of just installing. You actually get a lot of freebie commands like 'yarn why,' that tells you why you have a particular dependency or you could do 'yarn check.' It was just a lot more helpful.

I've been wanting to contribute to open source for a while so I did a little bit of work before then but the community was really encouraging when I first try to solve and pick up some first contribution bugs off of their backlog. At the time, they were pushing for 1.0 release so there's a lot of excitement about what Yarn will be and all the new features will be adding. I kept trying to pick places where I felt like I could be of help and then, Christoph, the manager of the Yarn Team and I believe [inaudible] team as well, reached out and asked if I wanted to build the selective resolution feature for them. I was like, "Yeah. I'll give it a go." Then I did it.

CHARLES: What is selective resolution?

KAYLIE: Good question --

CHARLES: Because I use Yarn every day and I've never heard this term.

KAYLIE: It's something that became available with 1.0 release, much like workspaces and most of the time, you're not going to need it but sometimes, you're using a library. That library will have a nested dependency that for some reason, has a bug or you can't work with that package or maybe you just want to dedupe the package so that all of your dependencies end up using one version of that particular nested package. Selective resolution is like a way for you to override other libraries dependencies --

CHARLES: Oh, that's really cool. I like that because I've had that happens to me where I want some version of a library that has got a bug fix and yet, some other library that's depending is requiring this library and they're getting the old version and I'm like, "Nah!"

ELRICK: That was happening to me last night. I wish I knew about this.

CHARLES: Yes, seriously.

KAYLIE: Yeah. Before this --

CHARLES: I'm glad we had you on justifying about this.

KAYLIE: Awesome. Before this, you would have to file an issue with the original maintainer and maybe, it'll get fix. Maybe it won't or maybe you're stuck on the old version and you can't do anything about it. We had a really similar issue with the PhantomJS package, where we wanted to use a next patch version with a bug fix but then, something that was requiring it wasn't letting us use it. It works. I verified it. It seems like Facebook started using it as well so it was a pretty rewarding to work on that.

CHARLES: That's exciting. I also plan on using it the next time I encounter this.

ELRICK: To get this feature, is this a flag that you have to pass? How does it do the selective resolution?

KAYLIE: As long as you're using Yarn 1.0 or above, you define resolutions field inside of your package.json, like how you would define that dependencies, you also have extra [inaudible] resolutions. On the left hand side, you put in a glob pattern that you want to match and if you want to match all packages, you just enter the name of the package. Then on the right side, you enter whatever version or path that you wanted to match. It could be a file or a GitHub link or it could be a version or a range or whatever you want.

ELRICK: Nice.

CHARLES: That's fantastic. Now, you mentioned that as you were coming to work on this, you were looking for features to work on but the community actually did a very good job of drawing you in and getting that contribution from you, which is actually pretty amazing when you think about it. I think a lot of open source projects, either flourished or failed by their ability to attract contributors. What was it that was particularly inviting?

KAYLIE: It was a combination of things. I think one thing is they tried to point you to the actual code like if you want to submit a PR, then this is where you would get started. I actually started doing that myself on some of the issues. Everyone loves PRs more than issues so I think giving people filing the issues, some, I guess empowerment to try to fix it on their own, I think is great. They also have a Discord Channel for devs to talk about any questions that they might have, how to set up their initial dev environment, to test things on Yarn. Also, they were really nice. They were really thankful when I posted a PR or commented on the fix. They also use a lot of emojis, which helps.

I think I personally found it really rewarding because it made me a better developer. Before contributing to Yarn, I didn't know that much about Node. For me, it was just fun to learn more about like, "This is how something else works," and also, the codebase uses a lot of different linting configurations, which I hadn't really used before so that was a nice learning curve there as well.

ELRICK: For your initial time going to Yarn when you didn't know much about it, was the champion from the project that worked with you to get you over the hump or places that you were stuck or did you just have to kind of figured it out on your own or did you ask questions on the Discord Channel or in Slack Channel or anything? How did that process go initially?

KAYLIE: I definitely pace myself. I just picked up easier bugs on a repos if possible and BYK and Mel who maintain the project would give guidance, especially through PR comments and they also answer any questions on issues. If I ask questions like what's the right approach here, because sometimes you get a bug and it's not just a bug. It actually has to do with philosophy of like, what should Yarn do in this case. There be like really minor edge cases like maybe NPM does that in a particular way, should Yarn respect that or should it try to be better? Those discussions are, I think really helpful and interesting and understanding what's going on.

CHARLES: It's fantastic when the conversation just builds and you're learning stuff and then you actually feel like you came out with the best solution that was available to you at the end of the process.

ELRICK: You gain a lot of context about the project during those conversations.

CHARLES: Right.

KAYLIE: Yeah, what was good about those selective resolution feature was it was completely community-driven, even from the RFC standpoint which submitted by someone in the community and then it was implemented by me, who doesn't work in Facebook. I think that's awesome.

CHARLES: There's been a lot of thought that was put into the feature even before the implementation. That's always so critical to getting people's and giving them some specifications, some blueprint of what they're actually going to build. One of the things also that I wanted to touch on too is you mentioned before the show that they have a very unique take on the way they handle communication with the community, not just in terms of pull request but also in terms of issues.

KAYLIE: Yarn issue count is around 800 --

CHARLES: Boy, that's a lot. I can feel daunting when you look at that, right?

KAYLIE: Yeah, but it's actively [inaudible] project, a lot of people are passionate about it and there are bots that other projects use to just automatically close issues when they're not active. But something that BYK told me was when issues are closed, people take it somewhat personally and we just want to make sure that there's like a human touch to it and we, at least get to the issues without just automatically closing them. I really respect that.

I think even when people are really frustrated, the maintainers never really lose their shit. They're always very graceful and when someone is acting outside of the standards of Code of Conduct, there's a gentle reminder. So far, all of the interactions that I've seen to the issues have been really constructive, rather than being like, "Sorry, we're not going to work on this." It feels like it is a community project and people care about how it's being used. It’s actually not easy given all the different operating systems that Yarn has to accommodate. It's like a pretty low level tool so I give the guys a lot of kudos for handling that.

CHARLES: Eight hundred issues means that with a human touch, that means 800 people have to actually respond to those issues and usually those 800 people are not actually 800 people. They're like 10 people or some number significantly below 800. How do you attack them to try and make sure that they're responded to in a timely fashion?

KAYLIE: I think issue tracking is the hardest part of open source. I think it's in the order of issue tracking and then reviewing PRs and then submitting PRs because writing code is writing code but understanding other people's code is more difficult and understanding other people's issues and what the bug is, actually is even more difficult. You don't, obviously have their exact dev environment so sometimes, it's hard to repro.

I think we do a lot of things that a lot of other projects leverage, which is we label the issues, we define priority if it's actually impacting a lot of people or if it's a critical bug like you can install a package versus maybe a warning message that could be tweet. Then we kept a board, like a GitHub board to track issues when we were heading for the 1.0 release. I'm not sure if we're still doing that but that helped to a degree. Other people from the community, not just the maintainers, jump in to try and help out an issue, which is probably one of the best things. Then when we merge pull requests, we close the issues that have been referenced.

CHARLES: Yeah, it's really wonderful when that thing happens. I'm so curious like how to [inaudible] because I know that it's so disheartening when you're working on a project and you file an issue and maybe, it doesn't even go answered for one day, two days or two weeks or you submit a pull request and no one's even commenting on it. It can be really disheartening and kind of make you question the viability of the product itself, especially if you see a lot of activity elsewhere. On the one hand, I also understand that the maintainers are human and they probably have a lot of obligations. I know that I've got a lot of projects where I let the issues languish and I even have one that I'm using where I can't get any response and it's just so frustrating.

KAYLIE: Yeah, even Yarn is definitely not perfect. There are definitely issues that sort of go buried. You could add us at YarnPackage/Core. That pings all of us and the core team. You could call out specific people but that's a tough one.

CHARLES: This isn't with Yarn, by the way. It's a totally separate project. It's fantastic when there's that basic acknowledgement. It makes such a difference in people's perception of the entire enterprise. Looks like we're actually approaching time. If we want to give a special shout out to any talks you're going to be giving, any events that you'll be at or --

KAYLIE: Actually, Ben on my frontend platform team is speaking at Nodevember about Next React. I think that's where the shout out. And then Christoph will be speaking at AgentConf in Austria, I think in January about the future of dev tools. I think both things are [inaudible].

CHARLES: All right. Fantastic.

ELRICK: Any slick future features coming out in Yarn that we should know about?

KAYLIE: I'll keep you updated.

CHARLES: She's sworn to secrecy.

KAYLIE: Yeah. I'm not sure what I'm supposed to [inaudible] yet.

CHARLES: I could hear the pause of conspiracy in your voice. Thank you so much, Kaylie for coming on the show. This has been a really wonderful conversation that's just gone all over the map and those are my favorite kind. Thank you very much.

KAYLIE: Thank you so much for having me. This has been a lot of fun, guys.

CHARLES: Well, if people want to get in touch with you, how would they go about doing that?

KAYLIE: I'm a rarity, where I don't have Twitter. You can email me, I guess.

CHARLES: File an issue on Yarn.

KAYLIE: Yeah. I'm KaylieEB on GitHub and KaylieKwon@Gmail.

CHARLES: As always, you can get in touch with us. We're at @TheFrontside on Twitter or just drop us a line on the email. You use that on occasion too at [email protected]. Thank you, Kaylie. Thank you, Elrick and thank you for listening.

Dan Gebhardt: @dgeb | Cerebris

Show Notes:

Resources:

Transcript:

CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode 87. My name is Charles Lowell, a developer here at the Frontside and your podcast host-in-training. Joining me today in hosting the podcast is Elrick Ryan. Hello, Elrick.

ELRICK: Hey, what's up, Charles?

CHARLES: How are you doing today?

ELRICK: I'm doing great.

CHARLES: Are you pretty excited?

ELRICK: I'm very excited for this podcast because this is a topic that I've heard a lot about but don't know much about and it just seems so awesome that I'm just very stoked to hear all the details today.

CHARLES: Yeah, me too, especially because of who's going to be giving us those details, he's one of the kindest, smartest, most humble and wonderful people that I've had the pleasure of meeting, Mr Dan Gebhardt. Hello, Dan.

DAN: Hey, Charles. Hey, Elrick. Thanks for having me on. I really enjoyed listening to this podcast. It's nice to be part of one.

CHARLES: It's good to have you finally on the show. We talked over chat and we talked over email and we meet every once in a while and conferences and it's great to get to share more widely some of the great conversations that always arise in all of those contexts. For those who don't know you, you are a founder at Cerebris and that is your company, which is involved very heavily in a lot of open source projects that people are probably familiar with. One of them that we're going to be talking about today is JSON API. I bet most people didn't know that you are one of the biggest driving factors behind both the specification and several of the implementations out there.

DAN: Yeah, that's been a pretty core focus of my open source work for the last few years. Actually JSON API Spec, which is perhaps a somewhat confusing name for those who aren't familiar with it. It was started by Yehuda Katz in almost three and a half years ago, I think now and it hit 1.0 a couple years ago and has stabilized since then and we've seen a lot of interesting implementations on top of it. There are some exciting stuff that's actually coming soon to this Spec that I'd like to share with you guys today.

CHARLES: To give us a little bit of context, why? What pain am I experiencing that JSON API is going to solve or it's going to address or give me tools to deal with?

DAN: One of its prime motivators is the elimination of bikeshedding. There's a lot of trivial decisions that are made with every implementation of an API and JSON API makes a lot of those decisions for you about how to structure your document, how to include relationships and lengths and metadata in a resource, how to represents relationships from hasOne/hasMany. Even polymorphic relationships have a type of that data. JSON API has opinions about all these things at the document structure level and it also has opinions about our protocol usage, how to use HTTP together with this media type to make requests and for servers to return responses, how to create a resource, how to add resources to relationships and things like that.

CHARLES: Also, it's not just this is a serialization format. It's very much like also delving into the individual interactions and how they should be structured, more about the conversation between client and server.

DAN: Yeah, in that way, it is somewhat unusual as a media type that covers both.

CHARLES: Can you dig into that a little bit because I'm very curious? Something made my ears prick up was when you said, it tells you how to, for example add relationships to a resource. What would that look like?

DAN: A lot of the influences behind JSON API are hypermedia-related. It's influenced by RESTful principles and includes a lot of hypermedia aspects. One aspect is how a resource represents relationships in terms of the data in document, the type and the ID that specify a linkage to that another resource in the same document but it can also include links to discover those relationships. There's a self-link for a relationship and a related link for relationship and the self-link will return the data for that relationship in the type/ID pairs.

The related link will return the related resources. The Spec doesn't have strong requirements or any requirements about URL usage but instead, it describes where to find resources through these hypermedia links. If you want to say, add records to a relationship, you'd follow the self-link for that relationship when that was returned with a resource. Then you would send a post to that endpoint and you would include the relationship data in the terms of type and ID pairs. It gets down to that specifications so that removes the ambiguity of how to interact with these resources and mutate them and retrieve them.

CHARLES: I see, so is there an idea then that you are going to explicitly model the relationships as individual resources? Or is that the recommendation or the requirement?

DAN: The link for a relationship would point to an endpoint, which would then model the relationships that are represented that endpoint, so to say just to speak a little more concretely because certainly, it makes some simple concept sound a lot more esoteric than they really need to be. Let's just talk about an example. Let's say, we're talking about articles and comments and maybe an author. Let's say, you've fetched a collection of articles from an article's endpoint and within the article resource, you would have a relationships number, which would include comments and then comments could have links, which one of the links would be a self-link and a related link. The related link could be followed to then retrieve all the comments for that particular article.

You could also, if you wanted to add a comment for that article, post to the self-link for that relationships. You post to that whatever endpoint that is specified. Maybe it's 'articles/1/comments.' It could be anything that you want. Now, the Spec does have some recommendations to make everything fit nicely in terms of the URL design patterns and such but those are not, by any means required but having those recommendations just eliminates more bikeshedding opportunities. We find it that people who are gravitate towards the Spec really appreciate having a lot of these trivial decisions made for them so even if we don't want to come down and be hard line about requiring those particular answers, we can at least provide some guidance for how things can work together nicely. There's a whole recommendation section on the site for things like URL design patterns.

CHARLES: Right, so things that aren't prescribed but these are best practices that are recognized.

DAN: Yeah, exactly.

CHARLES: A question then that comes to mind, it sounds like JSON API solves a lot of these bikesheds or just kind of comes in and takes one side or the other for modeling both the resources and the relationships between those resources so there's the... I don't want to call it a schema but the boundaries around which resource are very clear and where they live and how they connect together.

I was hoping we could maybe contrast that with some another approach, which is also become very popular and that's the GraphQL approach where you're essential assembling views at runtime for the client. It's very easy to marshal the data that you need to present to your view because you've got only one endpoint, as opposed to having to coordinate between them. I can understand the appeal of that and I was wondering if you have any insight into what the tradeoffs are between the systems and what are some of the capabilities that one can do that the other can't.

CHARLES: Yeah, sure. I'm glad that you brought that up because I feel like GraphQL has become a real juggernaut, at least because of its marketing. It's very effective in being marketed for its use to developers and it's capabilities versus REST, as if a RESTful system can't possibly achieve the same outcome or the same efficiency. I'm glad to compare and contrast the two. To be honest, one of our short term goals is to better tell the story on the JSON API site, which was always a kind of a more technical spec-y site and a marketing site. That hasn't really helped its uptake as much as it could as some of the GraphQL sites are very sleek and polished. Anyway, let's get down to it.

GraphQL allows you to basically define the data that you want for a particular view and that can bring together multiple related resources. It defines a way to specify exactly which fields you want in that graph of resources. We'll just stick with the articles, comments and authors example. You can specify that you want a collection of articles and perhaps the comments-related to that and the authors and you could have it all assembled in a single response.

JSON API also allows you to do just that. It allows you to make requests for multiple related resources to constrain the fields that are returned for each resource and to include all of these related resources in a single document. The main difference in the representation is that JSON API requires that resources only be represented once in a single document. GraphQL may have repetition of resources throughout the document that's returned. For an instance, your articles that may nest authors and those authors like Charles Lowell, may have written three of those articles and that representation of that author is going to be repeated in that JSON API compound document, which is a term for document, which has a primary dataset combined with related resources.

That single author would only be returned once as related resource and the linkage between the primary data and the related resources would be established to type/ID pairs. Instead of having the author represented three times, the same type/ID pairs would just be providing that linkage to the same author and that author resource would only be represented once.

This happens to be ideal for client-side applications that number one, basically want to minimize the size of a payload that sent. Number two, don't want to after-handle repetition of data by doing extra processing of pushing the same record multiple times into a memory store that is keeping that data. I think that GraphQL is well-suited to applications that request data and display that data pretty much as returned. There is no intermediate holding onto that data in, say a memory store for later access. Basically, it lines up well with a component library link React, which wants to display that data that's returned from the server. If it wants to display that collection again, it will simply request that collection again and pretty much throw away the data once it has been rendered.

CHARLES: I can see that. Dan, you and I might be some of the only folks who remember. I don't know if you ever did any Microsoft Access Program.

DAN: Yes, I did, believe it or not.

CHARLES: Doesn't it feel a little bit like the Access pattern all over again, where you have your components requesting data from basically, constructing a query and requesting it --

DAN: Yeah.

CHARLES: And then throwing it up on the screen.

DAN: You're going deep there but I do remember that. Definitely, there is that same paradigm.

CHARLES: It's really powerful.

DAN: It is and it's pretty accessible too because it's a direct representation of what you've requested and there's no intermediate processing. I guess the question is, whether that intermediate processing provides some value. Actually, holding onto that data provided some value because as far as I'm concerned, GraphQL is great for that rendering of DOM data, where the data has no meaning except outside of the rendering. But if you want to actually have models that have some intelligence about that data, then you want to use a store to keep those models in and you want to be able to reuse those models for other purposes.

CHARLES: What might be an example? What’s a concrete use case that we can ground this discussion?

DAN: I would say that the big one is offline. You simply can't have just DOM data that's useful in any way in an offline application or an optimistic application, where you are doing some things client-side and only say undoing them if a request fails. But if your data is DOM and only structured for a particular view, then all you can do with that is redisplay that view. But if you understand the schema of your data and that data is available in a store, then regardless of whether you have a network connection, you can actually display that data in different ways. If that same article shows up in a collection in a list, you could also display that article on its own in a different format with more fields. If you want to, say allow editing of that data, you could allow for an editor when your app is offline. Allow changes to be made to that data and then redisplay it because you understand the fields that are in that data.

CHARLES: Right and then at some point later, then spool those changes back to the server.

DAN: That's right.

CHARLES: It almost sounds like, ironically if a system like JSON API where you have very concrete boundaries around each of the underlying resources in your data model, it allows you to essentially do rich-querying on the client and not just the server.

DAN: Yes, that's absolutely true.

CHARLES: Because I feel like what you just described to me it's like, now we have some sort of store over which we can map all kinds of different queries to our own liking and there's no dependency on the server.

DAN: Yeah and if you just want more web app to be pretty much a view representation of what's on the server and without additional intelligence, then GraphQL really lines up well with your needs because any extra processing you're doing is just not valuable to you. But I think a lot of the really interesting things being done in client-side applications are where your client application is pretty loaded with a lot of intelligence and you're out there autonomous and able to make sense of data. In that case, then thinking about the data only as it pertains to views is not nearly as powerful.

CHARLES: Right, so you could do something like that with GraphQL but then you would have to, essentially structure your queries such that they drew the boundaries around the individual resources anyway, rather than composing them on the server. You'd have to query them discreetly into a store and then run your local operations. Then I guess at that point, it's like what are you doing?

DAN: Yeah, you're still doing the extra processing of handling the repetition of any nodes that repeat and such. That's just extra processing you have to do but I agree that you certainly could structure your GraphQL queries to return data that is then loaded, say to a store that really has awareness of the data types but I don't think that is --

CHARLES: But then you're defeating the purpose, right?

DAN: Yeah, it's not its selling point and it's not its strong suit.

CHARLES: You've done a lot of work on the JSON API Spec. JSON API allows you to fetch discrete resources and their relationships but still, keeping one representation of each resource in the payload so it's optimized for wanting to do client-side processing and have intelligence based on these entities, which are in a store. You actually maintain a fairly mature, at this point, framework called Orbit, which helps you do some of these things.

Now, what Orbit does today and I understand that you've got a lot of new features that are really exciting, that are coming down the pike. Before we get into those, what is Orbit and what do you use it for and how does it use JSON API?

DAN: Orbit is a data access and synchronization library, which sounds sufficiently vague because it has a lot of low-level primitives for structuring client-side. Also, actually isomorphic can be run on the server and nodes so it's not even only used for client-side purposes but that was its original purpose. The abstraction that it includes are allowed for synchronization of data changes across multiple sources of data. Source of data might be represented by, say a JSON API server, an in-memory store, an IndexedDB database in your browser, a local storage, all of these sources of data can support an Orbit interface, which provides their access to their data and also broadcasts changes to that data.

In order to coordinate the changes across multiple sources, say to back up all of your data that's in memory to IndexedDB source, you can observe the changes on one source and then sync those changes up with another. For instance, you want to structure an offline application which you have been in-memory store, which uses client-generated IDs, which then syncs up with a backend JSON API source and every change that gets made to the memory store needs to be backed up, you could configure multiple coordination strategies between the sources to make sure that the data flows so that every change that is made to the store is immediately backed up to IndexedDB. If it can't be backed up, then it fails.

You can add some error handling and then when you're online, you can then also sync those changes up with a backend so you're basically pushing those changes that are local to a remote store and you're not slowing down your offline app, which you're communicating with optimistically and then only handling, say synchronization failures when there is a problem. In order to handle those problems, Orbit sources are very deterministic about their tracking of changes and they provide get-like rollback capabilities so you can look at the history of changes to a particular source and reset the history to any point there and basically handle conflicts and merges in a very get-like way.

Often I use cases, the primary driver of Orbit's whole architecture, I realized that it needed to be able to give you the tools to handle any conflicts that happen when changes get sync up. Also, give you the different tools to model all the different places of data is kept in order to support the offline mode. That's a kind of a broad overview of Orbit.

There's a new guide site, OrbitJS.com for those who want to dig a little deeper into it. The data is structured in the JSON API format internally to the store and the standard operations are very much influenced by the standard JSON API protocols that are allowed in the base Spec over creating records and removing records and all that crud for both records and relationships. That's where JSON API comes into Orbit.

CHARLES: Right, I see. The primary use case for Orbit is offline. Is that fair to say?

DAN: Yeah, that was the primary driver, although it's just not the primary --

CHARLES: It seems like you could use this in a lot of places where I might use Redux or something like that, like on the server to model... I don't know, a chat app.

DAN: Yeah, definitely.

CHARLES: I have a bunch of different information streams coming together and how am I going to merge them and make sense of them.

DAN: Yeah, in fact, that it's primitive level. Orbit has essentially an async redux-like model for queuing up changes and applying those changes. The change sets are all immutable. There's actually a lot of immutability use here throughout the library. In order to ensure that the changes that are applied are tracked deterministically, we just can't have those changes mutating on us. There is definitely some overlap with Redux concepts in terms of the general tasks or action concepts in Redux but instead of Redux's synchronous approach, everything in Orbit is async.

CHARLES: What does that mean? Redux is synchronous in the sense that there's a natural order to all actions. For those of us familiar with Redux, are you saying it would be like a store where actions can be dispatched at any time or is it more like, I've got multiple stores happening and I need to resolve them somehow so each one is synchronous? How can I make sense of that?

DAN: In Redux, the actual application of an action is performed synchronously.

CHARLES: Right. You can have asynchronous processes but there is a natural order to the actions that those asynchronous processes yield and then those are applied synchronously to the Redux Store.

DAN: Yeah. To compare and contrast Orbit and Redux, I guess you'd first have to say there's a primary difference of --

CHARLES: I think there are a lot of people are familiar with Redux. I think it's not so much to compare and contrast it but just to use it is as an analogy of like, "Here's how it's the same here. Here's how it's different," because that's compare and contrast.

DAN: There you go.

CHARLES: But not in terms of evaluating them but it's like, "Maybe I should be using this instead."

DAN: Right, they are sort of on different levels, although there are some primitives in Orbit to it and it's shipped across multiple libraries. There are some primitives, I think that could be useful outside of the main Orbit data application. Anyway, the way that Redux state changes are applied, the function is synchronous is all I was getting at and on Orbit, every state change that applied to a source is asynchronous so the result is never applied immediately. You'll always get a promise back and you'll never have that application happen immediately. That's one clear distinction.

Another is that a redux has a big singleton global state for the entire application. Orbit very much has a model of state per source so there can be any number of sources in a particular application and the source might be an in-memory source or might represent a browser storage in XTP or might represented a socket that streaming data in. All of these have state at different, temporarily distinct state that even if they all converge to a common state, the Orbit models separately so that there's a set state per source. I'm just contrasting the global apps state that exists in Redux with the per source state in Orbit.

CHARLES: It sounds like there's nothing that would be fundamentally incompatible of using Orbit really in conjunction with Redux, where Redux is kind of a materialized view of all of your different data sources presented as what you're going to render off of, right?

DAN: Yeah. You could use it in a similar way to Redux Saga, where Orbit fills the role of Saga, where it's doing the asynchronous actions that results flow back into the Redux state.

CHARLES: I'm just imagining having one big global atom, which is your Redux store and now I'm saying, prescribing this is an optimal architecture but I'm saying, one way it could work is it picks and chooses and assembles off of the different sources as new data becomes available. As the states change for those sources, it can be integrated into a snapshot state, which is suitable for rendering or provides one view for rendering.

DAN: Yeah. You're basically talking about the in-memory source, perhaps merged with other applications state, which is not so resource-specific and that is possible to model.

CHARLES: What I think I might be hear you saying is you could also just use another source which is the merge itself.

DAN: Yeah. I'm not sure how much we want to continue this thought exercise because the architecture becomes almost not something I'd recommend. But I would actually like to explore how Orbit and Redux could be used together optimally. I played around a bit with Redux but I have not written a full-fledged application with it, other than a [inaudible] location. I definitely defer to you for Redux best practices and such and how people are using it in real world applications but I'd be really interested to talk that over again soon.

CHARLES: Well, I just certainly don't count myself a Redux expert, although we have developed some applications with it. We'll put that on the back burner or something to explore it later. I will say this, I find Redux to be both wonderful and terrible, kind of in the same way the Java is both wonderful and terrible. We’ll leave it at that.

DAN: Okay.

ELRICK: That was going to be my question. This is why I was very excited to hear about today was Orbit because I've heard so much about it. In terms of the implementation of Orbit into an application, what would that look like from a high-level? Has anyone used Orbit in the production app? Have you built any apps using Orbit?

DAN: Yeah, definitely. There are people using Orbit with React, with Vue, with Angular and with Ember and there's an integration library called ember-orbit which makes Orbit usage really easy in Ember. In a lot of ways, working with ember-orbit feels a lot like working with Ember Data but it allows a lot more flexibility. I suppose one of its strengths and weaknesses is there's a lot of configuration that's possible because there's a lot of possibilities. The internals are exposed of how data get synchronized so you can define your strategies and sync up different sources.

In terms of how it's actually used in an application, you'd start by modeling your data in terms of the resources that are in the application. You'd have a schema that defines your articles, your comments and your authors, just to keep that example going. Then that schema would be shared among all the sources in your application. You would have one source, say that might be the in-memory source and another source that is the representing a browser storage so you could, say swap out either local storage source or an IndexedDB source and use either one to provide that backup roll.

You would declare those sources, you connect them to each other with strategies so that, say when memory storage changes, you would then sync that change to the browser storage source. Then you'd have back up and you'd be able to then, refresh your page and view the same data you were looking at before. Now then, if you probably want to wire up a remote source so that you're communicating with the servers so you bring in JSON API source and you would then set up a new strategy for working with that. You have to decide like, "When my memory storage changes, do I want this change to happen optimistically or pessimistically?" By that I mean, "Do I only want it to appear successful if it's been confirmed by the server."

Depending upon whether you want to be optimistic or pessimistic, you setup your strategies a little differently. If you handle this change pessimistically, you wanted to block success on the successful completion of pushing of that change to your remote server. You have the set of tragedies that define the behavior of your application and then doing your crud operation is probably pretty much directly with your memory source. Then if you wanted to, say do an edit in a form, you might fork the store, now the store keeps its data in an immutable data structures. That forking that store is very cheap so you don't have a bunch of data that's copied. You're just keeping a pointer to that and getting a new pointer to that same immutable data structures. Every time they get changed, there been new references.

There's an immutability under the hood but you're pretty well insulated from the annoyances of working with that immutable data structures. At that form, you make your changes, you then merge your changes back, you'd get a condensed change set of operations that then can flow through your strategies. It flow through to your backup source. It could flow through to the back to the server. I think it would feel pretty familiar for users of Ember Data because there are a lot of the API influences came from that library. But obviously, people are using just plain Orbit with other libraries, with other frameworks and finding it useful there but it definitely involves a little more configuration up front to do all that wiring that might be more implicit in library like Ember Data.

CHARLES: I understand that before we go, there is some pretty exciting new things coming in Orbit. Do you feel like you're ready to mention a couple of those things or they've been kind of mixed in with the conversation?

DAN: Let's see. I have the guides up, which I mentioned, which is pretty new in the last couple of months. In the last year, we did a rewrite and Orbit is now completely in TypeScript and there are no external dependencies. For a while there, I was using RxJS and observables internally and immutable JS so there's now an internal immutable library. It's lighter-weight with fewer dependencies now. I'm excited about that and finally feel like I can recommend people digging in with the guides that are up. I'm hoping to get up the API docs soon.

I will say I'm excited. I just got back from a retreat in Greece. Séb Grosjean who owns the company, BookingSync does this amazing thing with the Ember community, where for the group that's working on Ember Data, he invites them every year to come to his family's place in Greece. He grew up working with his family on his rental properties, which was the inspiration for his company, BookingSync and said, "This is a fantastic opportunity that for us to get together and collaborate in a really nice place," and I had a really productive time this last week.

This is the very first time I had gone. It was just fantastic and I worked with the Ember Data team. Igor Terzic and I spiked out some interesting collaboration between Orbit and Ember Data so I'm really looking for it to see where those go and hopefully, we'll see a little bit more Orbit, either directly or just through influence appearing in Ember Data. I'm looking forward in working more closely with the Ember Data team. We'll see what comes of that.

CHARLES: Yeah, I, for one am very excited to see it. I'm resolved now. I'm just looking at these guides. These look fantastic and I'm resolved to give Orbit, at least a try here, either in some of our applications or maybe trying to spin up some new ones and have it the basis for some of ideas I've been playing with.

DAN: That would be awesome and there's a [inaudible] channel, which I hang out into if you have any questions, if anyone out there does.

CHARLES: Before we go, if anyone is interested in JSON API, is interested in Orbit, is interested in Cerebris, we mentioned a lot of things that in one way or another, map back to you. How do we get in touch to find out more about these different entities/projects?

DAN: I'm at @DGeb on Twitter. My company site is Cerebris.com. Also check out, OrbitJS.com for the new guides. Reach out to me. I'm on the Ember Core Team so I'm also hanging out in the Ember community Slack, depending upon what you want to talk with me about. I'm in all these different places so I love to hear from you all.

CHARLES: All right. Fantastic. We'll make sure that we put those in the show notes and I guess that's about it. Do you have anything else you want to leave folks with, any talks, papers or big news coming around soon?

DAN: Something that we didn't really get a chance to talk about today, which I'm really excited about is JSON API operations, which is an extension to the base Spec, which I'll be proposing very soon. There's a future to the JSON API once it hit 1.0 a couple of years ago. It didn't just stop. We're looking at different ways to extend the base Spec and use it for different and interesting purposes. JSON API operations, I think one of the most interesting ones.

The idea is basically to allow for multiple requests that are specified in the base Spec to be requested in a batch and perform transactionally on the server so the Spec will define how would each request gets wrapped. Each operation very much confirms with the base Spec concept of a request. For implementations, there's a lot of opportunity to reuse existing code for how to handle each particular operation but to provide a whole new set of capabilities by allowing you to batch them together and process and transactional it because it just unlocks a ton of different things you can do, all based on the same base concepts from JSON API. I'm really excited to have something to announce soon about that.

CHARLES: That sounds like it might solve a lot of problems that are always associated with those things. It always comes up. What's our batch API look like? I don't think I've been on a project that didn't have a months-long discussion about that. I ended up like kicking it down the road and I'm just flumping something in place.

DAN: Yeah, all those messy edge cases where people figure out how do we create multiple related records altogether in a single request and people do it ad hoc and do it with embedding and such and want to standardize that in the same way, that we've standardized the base operations.

CHARLES: Well, that is really exciting, Dan. I wish you the best of luck and we'll be looking for it.

DAN: Thanks a lot. Thanks for having me on, guys.

CHARLES: It was our pleasure. Thanks. With that, we will say goodbye to everybody. Goodbye, Elrick. Goodbye, Dan. Goodbye everybody listening along at home. As always, you can get in touch with us. We're at @TheFrontside on Twitter or you can see our website at Frontside.io or just drop us a line at [email protected]. Always love to hear from you with new podcast topics, anything that you might be interested in so look forward to hearing from you all and see you next week.

Alex Matchneer: @machty | FutureProof Retail

Show Notes:

Charles and Alex Matchneer have a great discussion that centers around routing in Ember.js: what they want to see in a router, what problems it solves, what’s wrong with the routing solutions we currently have, and what the ideal future looks like in respect to routing.

Resources:

Transcript:

CHARLES: Hello everybody and welcome to The Frontside Podcast, Episode #86. My name is Charles Lowell, your developer here at the Frontside and podcast host-in-training. I'm flying solo today. It's been a while but that's okay because I've got a really fantastic guest on. Actually, we debated this at the beginning of the show, whether this was the third or the fourth time he's actually been on but no times are too many so hello, Alex Matchneer. Welcome back to the podcast.

ALEX: Thank you. It's great to be back.

CHARLES: You're still at the same place that you were the last time.

ALEX: Yeah. Still working at FutureProof Retail. I'm still working on bunch of mobile ember-cordova apps and that's definitely occupying on my time.

CHARLES: Nice. Because FutureProof Retail has a large hardware component and we were doing a series on IoT, we were originally going to have you on the show to actually talk about that experience of what it's like to be a part of a startup and develop software that's going to be running on a bunch of devices and the unique set of problems that poses. But in the pre-show, we decided to scrap that because there's actually a topic that we're both very interested in and you've been heavily involved in lately and might be a really interesting preview as to what's coming in the Ember community and at large.

Today we're actually going to go back to talking about the same subject that we talked about in our first podcast, which is routing: what we want to see in a router, what problems does it solve, what's wrong with the routing solutions that we have today. Talk about what that beautiful, ideal future that we want to live in looks like with respect to routing. You've been thinking about this a lot lately. What have you been thinking?

ALEX: I'm an Ember core team emeritus and back when I was on it and I'm a lot more active, I did a lot of work on the router, particularly with how it handles asynchronously loading data when you click on links and go to different sections of your app. I spend a lot of time over the last three or four years figuring out the nice patterns for what you actually want to use if you're building out lots of Ember apps. Then kind of around that time, right after landing some cool stuff and some not cool such us query params, which has been a challenging aspect, I start working at this company FutureProof Retail that is like 90% of the Ember work that I do there is in mobile apps.

We use Cordova so we're basically running these apps inside a web view, inside either iOS or Android so that we can stay with the technologies we are most familiar with, such as JavaScript and CSS and HTML and build apps using that. We can use Ember to do that. What I found was that I couldn't really apply a lot of the same patterns, all these nice conventions that Ember router gives you. I couldn't really find a way to map that onto what I need to build in mobile apps and there's a few different reasons.

I got really busy with the startup, just trying to build these things and kind of went off the happy path where I really just couldn't find a way to make it look like an Ember app. One of the nice things about the whole points of convention over configuration as this sort of Ember and Rails philosophy is that, one of the benefits is that if you know Ember and know Rails, you can drop into someone else's apps as long they're following these basic conventions and immediately know how to be productive and know how it's structured, know how to make a change to it and have it maintain a convention and not just have everybody who's using some framework build these totally different apps from each other that have no shared conventions and whatnot.

Everyone is supposed to be able to learn from each other, grow with each other as long as they stay with these conventions. I couldn't really find out how to stay within Ember conventions and build this mobile apps. For a long time, I just didn't really contribute too much to the Ember router at all. I kind of fell out of touch with how most people are using it because most people are building these desktop-centric apps and here I am working on these mobile apps after three years.

CHARLES: What are some of the specific use cases that were just impossible to, or not impossible but presented a challenge?

ALEX: The first one is which is I think is actually one of the easier problems to solve but still some challenging is that you want something that's called stack routing or stack navigation in a mobile app, which is if you're actually building a native iOS app or an Android app, they both have different names for how they provide you this. But you're thinking of things in terms of stacks. In Android, you might open another activity, which is a full frame of a page in your app and you can push it and then when you press the back button, which is built in in Android phones, it'll pop that off the stack and take you back to where you were.

In iOS, they give you a UI navigation controller and let you push and pop view controllers and that is how they want you to think about these applications. That is contrasting to what Ember makes you think about, which is go and define your static hierarchy of all the different places that you can be in an app. But with stack-based navigation, you don't necessarily know upfront all the different orderings of which frames are going to be pushed onto what and you might have situations where you want to be able to dynamically push, say an 'Add a Credit Card' page to where you are and maybe it depends on some data that's been loaded at some lower level in the stack and you can't model that as nested routes in the way that you might think about it in classic Ember apps. It's a different structure --

CHARLES: Now, when you say lower in the stack, I'm curious, if you're entered in aren't you... Oh, you mean... I see, previously in the stack. Okay, so lower in the stack so you're thinking like your current position is at the top of the stack.

ALEX: Right, yeah.

CHARLES: I see. Now, let me just clarify this in my own head. Your Ember routing structure is ultimately realizes a static tree but at any moment, you are entered into one path through that tree so you do have something resembling a stack. It's just is it the pathways that the ways that you can actually get nodes onto the top of the stack is you're limited because that can't be dynamic.

ALEX: Yeah, but even then, it's hard to describe what the difference is but the kind of stack that you're thinking of in terms of the classic Ember router map is more like you're in these different substates than you are different frames that you've pushed onto your --

CHARLES: There's a finite and fully enumerated set of next states.

ALEX: Right. To be very concrete, if you have a post route and then a post show route and then a comments route under that and these are all nested in a row, then if you're in the comments route, you are in a kind of hierarchical stack that might have loaded the post that you're looking at and maybe the post call-to-action above that and the comments for that but you're still in one thing. You've just expressed that one thing in terms of these substates so that every other state that's in the parent state can share the same data loading. That's different from saying, "I'm on this page and now, I want to push another page on it and maybe tap some of the data that has been loaded on previous pages." That's more of a navigation stack in a hierarchical substates stack.

CHARLES: Is the difference then, the data dependency? Because if you think of the Ember classic where you got the static tree, at least theoretically all of the data in the leaf nodes depends on the data that's above. It's not just being able to dynamically push stuff onto the new stack but it's also saying, you want to be able to push stuff that might have no dependency on the stuff further up and it doesn't need to be re-rendered if stuff further up the stack changes.

ALEX: Correct.

CHARLES: But sometimes it might.

ALEX: Right so there are a lot of corner cases that come out if you try to model this new way that a lot of corner cases have been thought out of if everything matched nicely to this hierarchical substate classic Ember stack but not for navigation. If you want to do something that's stacked routing-based, I've had a few different approaches. At our company, we maintain a suite of different apps that are sort of retailer or grocery-centric and the first one we did, which is more popular flagship one is Mobile Checkout, which is an app that lets you going to stores, scan items with your phone and checkout and skip the cashier line, which is great if there's huge lines and you just want to buy a little handful of things or maybe in your shopping cart. But that is like any other mobile app is really conducive to this step navigation approach.

Then we had to make a few apps after that such as like another app that is [inaudible] do a manual check then ordering app and other of handful things that you can imagine is might be used on a grocery store. I took the opportunity to like, "I don't really like how the routing turn out the main mobile checkout shopper apps so let's try different things." If you approaches, at least have their pros and cons without really feeling you're solving the problem and one is to maintain your own in-memory stack of where you were, every link to you, you might recall where you were and then use that logic in addition to what's in a URL to decide what transitions to make, which to use Liquid Fire for that.

But already, there's these weird growing questions like, "Why are you even using the URL? Is it helping you at all?" That was the main issue with the main app that we did. The other approach was to try and not even use any of the 'router.map' stuff at all. I use the router.map to basically just create one wildcard route. You can use normal Ember to use it like '*half' and that basically collects the rest of the URL as a param that you can use to do whatever you want with. I was using that to basically pass to another, which is internally used by Ember to do the stack-based parsing like grab a little bit of the URL and then parse the param for that then grab another.

Every time you could see your stack in the URL. That has its benefits but the worst part about it is that it's getting further away from Ember so any add on that you might want to use at Internet of Things in terms of which route you're in and has conventions like that you just can't use. I can't think of a good example at the top my head but it's like the further you get away from those norms, the less the Ember system can help you and on your own building your own framework.

This is all to say that I think I have enough experience at this point to bring home some of the things to Ember and I'm excited to get back into contributing to Ember with this one particular thing that I'm focusing on now, which is... I don't even know what to call it. It's like --

CHARLES: What does it do? The route stuff?

ALEX: It's route stuff. Actually, let me get into the other... That's what is tricky about stack routing and tricky to sort of, if you already have to go through a mental hurdle with thinking of the Ember router and as a stack of states or substates and you train your brain to think that way, it's really hard to take yourself out of it and realize that what you're trying to build with like a classic mobile navigation is almost looks like the same thing but it's really different.

The other challenging problem, which is specific to our particular app is that you wouldn't think of it as a very heavily server-driven app but if you're writing an application that at any point can get a message from the server like, "Hey, your status has changed," and that state is heavily coupled to navigation of where you're allowed to be in your apps for the state of some certain model, then you're going to have a really hard time, I'd say in modeling an Ember. I have a really hard time convincing people of this until they've actually tried to do it themselves, which is why I'm going off and just building things showing people.

CHARLES: You don't have to convince me because I think one of the biggest problems is the router is like the one non-reactive piece of Ember, which is unfortunate because it's essentially, what is the equivalent of the Redux store in a Redux application, where it's the state that drives literally the entire application and yet, any type of non-hash change driven updates, you have to manually manage. Every time that we've done it, it's been a problem and depends on what data, at that point you have to be very thoughtful because, at least from the highest level, if there's damage to a piece of the tree higher up, you need to realize those effects of that damage or that change all the way down the tree.

ALEX: Exactly. That is a great way of putting it. This is maybe a good time to mention this thing called ember-rideshare. I've had a really hard time describing these problems to people so I figured what I would do is write this blog a few months back, a little article called ember-rideshare. It's just a given name to the kind of app that still really hard to write in Ember. It's a mobile app. It involves stack routing but the other part is really difficult about it is this problem of the router being in a silo.

It is reactive but it's only reactive to that URL. Other things changes, they need to, like you said come in and patch up something else about the router in case you add some URL that is no longer able to present some model of whose status changed. That's an article on a blog that I can probably link to in show notes or something. When I talk about ember-rideshare, imagine using Ember to build Uber or Lyft and it's got just the slightest bit of the whole thing. The whole point of the app is to coordinate your client-side request of I want to ride with the server going off and doing a bunch of things and finding a nearby driver, displaying you bunch of driver locations and it'll show up. Then finally, find you a driver. It's a constant communication.

Throughout that point, you can sort of imagine modeling all the different screens as routes but the routes that are actually allowed to see at any given time are heavily dependent on what is the current state of the user's current ride. But you shouldn't be able to go to a route that says like 'cancel ride request,' if you haven't requested a ride in a million of these other things. If you're an Ember developer and you think that's an easy problem to solve, you're probably thinking, "I would use before model hook when I'm entering that route to check the state of the model," and if it doesn't make sense for the route of entering, I want to transition elsewhere.

That's fine. That's good if you're doing an app if the user is the one deciding where to navigate to. But then when you're on a route like that and then the server tells you that your ride is done, you can't still be on that route so you've got to have some kind of validations that is like, "This is no longer a valid route to be in. Is the user still in this route?"

CHARLES: "Where am I going?"

ALEX: Yeah. Before model doesn't really help you. It's this one-shot discrete event and you just can't capture all the different things. The ember-rideshare describes some of these problems a little bit more detail but that's the main issue with it. Like you said, what is actually missing about the router? Maybe it's reactive but it's only reactive to the URL, what about all these other things that are happening into your app? I think there's a handful of APIs in Ember that they're great but they're kind of siloed off in a way. If you want to make two different kinds of worlds meet, you've got to write a bunch of your own code yourself or you just have to do mentally going back and forth and being like, "I did this, so I can't use this kind of API."

I did a lot of work on the Named Blocks RFC, which previously there is silos between if you're passing blocks to a component versus data, you've got to think about them differently and all the ways that you might forward that data to a different internal component, if you want to build these composable, reasonable internals, you got to be kind of split-brain about it. I feel the same way about how the Ember router works. It's only good at dealing with stuff that has to do with the URL and you're on your own, if you needed to react to data changing. That's what I'm trying to fix. Does that correlate with your experience of working on Ember stuff as well?

CHARLES: Absolutely. I think that's a great way to put it. I think we've come to a consensus of the problem statement. I am curious to see a big separate query params. I'm going to throw that wildcard out there or maybe we should save it for later.

ALEX: Yeah, I definitely going to come back to it. If I say all this cool stuff and I still don't have a solution to that, then what am I talking about?

CHARLES: Right.

ALEX: Which to be honest, I haven't thought of every single possible thing. I'm doing the thing where I talk about it on a podcast that everyone can guilt me into really finishing it. I actually really think that I'm going to finish it. I'm very confident in stuff I'm working on. I'm very excited to bring it to people but it is not all 100% fleshed out and I definitely appreciate anyone's help to those interested, understands the nature of the problem and wants to help me work on some of this stuff and like that, in Ember community Slack or wherever.

CHARLES: Yeah, I'm really excited to hear it and see in what ways we might be able to contribute.

ALEX: Basically, the goal is to find some underlying primitives that can model the current behavior without mistake because obviously, we can introduce something that's going to break into Ember apps. Basically, to recognize that the URL is something that goes through multiple passes of transformation, to eventually become the thing that displays stuff on your screen, from the very foundation of it, and this is the actual mini-course of what Ember router does internally because it involves a few different libraries and maybe this is a re-hash from the podcast that I did with you guys but --

CHARLES: Can I just say that there are some things that the Ember router really does right, that are fantastic? One of those things is it baked in to every single piece of data. It doesn't do the stack but in that tree that it models, every single node in that tree abstracts away the asynchrony of that node. I think that's absolutely huge so you get both the dependency enumerated like these are the things that I need to marshal the data to render myself and it's implicit that it might take some time. I might need to draw on a couple of different things to actually assemble this data so the asynchronous nature is modeled up front and it's implicit and it's there every single time, which turns out to be the right thing.

The sampling that I've missed has been an excruciating void in all the other routing solutions that I've tried outside of the Ember community is that they just punt over asynchrony to you. You deal with it, not our problem and it's like, "Actually, that is the problem." Anyway...

ALEX: That's a great point because if the router doesn't help you with any of this stuff at all, then it basically means that every one of your pages that you might want to render after the fact, probably has to have some loading logic like if data is loading, show us spinner. Otherwise, here's all the data --

CHARLES: Yeah, if something happen wrong.

ALEX: Right and sometimes that is actually what you want to do. Sometimes you want to do these skeletal in UIs that looked like the page that's about to display but the date isn't there yet so everything is, regardless going to be wrapped in these 'if' statements, 'else' statements. I worked in ember-concurrency and some people are using that to basically move more of that loading into controllers, that's fine. If that's what you're actually trying to do and that's what you're opting into, that's a perfectly reasonable solution but most of times, chances are you're entering a route and you don't want to have to teach the entire template tree underneath it that has to handle all these different states.

There's these nice ideas that work in some cases and I'd like to make them work in more cases than Ember helps with and a whole loading all the promises and the model hooks and absolutely going into the loading state are really cool primitives that Ember is going to do for you. The other frameworks, they don't try to be opinionated. They won't do any of that for you. Sounds like you ran into that with some of your React stuff?

CHARLES: Yeah. I definitely did. There's just not much help when you actually want to model asynchrony. You can do it. It's pretty easy. You just implement the right hooks or model a series of actions, either with a Saga or Epic, if you're using redux-observable. But again, you have to assemble it by hand and you have to generate those abstractions by hand and you just want to have them at hand already and not have to worry about that.

But the advantage, though is that generally those ones that you do have at hand or that you generate are fully reactive. If new information comes that's germane to that particular leaf in the tree or that particular note in the tree, there's no difference between the initial state and the update state. Whereas, in Ember, you got your first shot and then that data is now at rest.

ALEX: Right. I definitely have been looking at React router, in particularly v4. I think it's all contentious for people to see it at first but being able to put things like in your render function, you can say, "If this data is present, something that's going to be past and be a prop or something," then show a loading spinner or otherwise, start matching these subroutes. That's really cool. That's expense that you can't look at essential map of all the states of your router can be in but that's also a real problem and if you can demonstrates that the state world is not in a separate silo than the routing world.

CHARLES: With great power comes a lot of bugs. You do run into a lot of things where you have rogue matching. You have random things that are inside your view tree that are matching against the route and they just render and you have to be very careful because it's almost the difference between blacklisting and whitelisting. I see what you're saying. It could be confusing.

ALEX: Yeah. I think it's definitely a tradeoff. I think if I had something like a match, I might have been able to maybe arrive at a stack routing solution a little earlier. I'm not sure about that. It's definitely something that could be handled by React router. I think one of things that React and React routers better at in general is that everything is, more or less a component that is more easily swappable or something else here. You're not going to have as many of those silos and I really do think, it went through a lot of churn and maybe, some people had trouble, maybe a lot of people, I don't know had trouble kind of following all the major versions. But I think React router Version 4 is pretty damn cool. I think there's a fullest realization of that kind of modular mindset.

CHARLES: I think the biggest problem I have with it, though is it requires the view tree to model your routing structure. That bothers me. I feel like you could do the exact same thing. You could have a way to express your routes, not necessarily with a separate routing file. I supposed you could do it with JSX or something but actually have it be kind of orthogonal to your view tree. The way you can model this dynamically updating thing that can match against anything and maybe, even express it all in one place. Although once you get a big tree, it could be hard to control that.

The part that I've come into most conflict and maybe who knows, maybe I just haven't used it enough, we've only got one application that we're using the router V4 on. But the fact that it's actually in the view tree, it bothers me. It's in the state objects. It's hard to adapt to Redux because that state is opaque. It's the routers controlling it and I would it to be not have to pass through React components but just be like, give me the firehose of the router state.

ALEX: Right. I love what you're saying. If I'm going to bring this stuff to Ember, I can't suddenly make it work like matching within the view tree. That's not what I'm working at or proposing here. All the stuff is basically to empower that firehose to respond to more things that can drives views and respond to them in a live way, not like a one-shot async validation, only when you enter.

CHARLES: Maybe this is what the problem that you're trying to solve and one of the things it's really nice to be able to match against anything inside the view tree is that Ember's rendering process of a route is very opaque. The process, by which an outlet gets connected, that's not something that you really have much visibility into. Is that a good statement of the problem?

ALEX: That's definitely part of it. You definitely have to go to the documents. I think it's telling that --

CHARLES: I've never done it. I don't really know how that works and I've written a lot of Ember code.

ALEX: How what works?

CHARLES: How the route gets rendered, like the mechanics around, which I understand how the route object actually, you makes the decision to render its template and do all that stuff. I know it as a user but I don't know the mechanics and I wouldn't know how to extend it.

ALEX: I'm not sure if the stuff I'd work on but it immediately make some of that stuff more clear. One of the goal or constraints is to really try and break down the silos. Whatever I'm about to propose bringing to Ember, I want it also be something that would be useful, possibly at the component or template or controller level, rather than just being this thing that lives only in the router's weird black box of logic that occasionally calls hooks that everyone knows about.

CHARLES: Right. In a sense, I'd say that they both suffer from that same problem. I'm curious to hear about the firehose.

ALEX: To actually get into what I think you're building here, we can dance around it all day and then we --

CHARLES: Just save it for the last 30 seconds of the podcast. That way there could be no --

ALEX: We're swapping JS for React router V4. Bye! It’s basically this. What's happening today is that you have a URL, it's going to be parsed in a way that you've tied it to via the router map file, which every Ember app has the place to go to see all the different places that you can navigate to an Ember app, which is great. You basically taught Ember how to break your long URL string into these usable bits and that's going to give you an array of these things that internally who cares what they're called but they're called handler infos and they basically say, "The first element of this array is named application. Every Ember app has one. It doesn't have any params."

The next one, it starts getting into what your URL actually is. Maybe it corresponds to the '/post' portion of the URL so that's going to be named 'post,' and that doesn't have any extra params either. Then there's this thing that is post show or something like that. That has a dynamic param because that's the part of the URL as like the '/123' and that corresponds to the post ID. It's basically, if you like thinking of things in terms of transformations or observables or mapping and functional transformations, that's taking a URL and turning it into an array of these useful POJOs of information. The goal is to keep transforming that into something eventually has enough data to display and templates and whatnot.

In this giant black box of the Ember router, it's going through those transformations and then it's going to go through this long series of using these params and this useful array of POJO information, start hitting hooks on people's routes to load data. Hit before model after model, redirect all these things to give tasteful names to all the tons of validations and checks that you might want to do. You do cool things in your before model hooks, check if the current user is actually an admin to prevent them from going into any '/admin' subroute. That's a really cool place to go and it's also a great convention.

If you're new in Ember app, you realize you can't go on this route. It should sort of click in your head and that sounds like they've got one of these redirect hooks to ensure that you're not going anywhere you're not supposed to go. All these things are really still to this day, extremely strong, well-designed, it went through many passes of review before it landed. I think they cater to a certain kinds of user-driven clicking around apps but they are extremely strong to this day.

I think the only thing that's missing is the smell. That example I gave like checking if the user is an admin, it's a bit of a smell that is not reactive. It's a hook. If it passes, great. You're in the route. It's not going to keep on checking that. What I want to do is basically, either in addition to or as an alternative to specifying these one-off model hooks or these hooks that you, not only really just fire one time, have essentially what is an async computed property or an async validation that is upfront about things it depends on.

Ember is going to be smart enough to constantly reevaluate these things as stuff changes. It can depend on not just URLs or URL parameters but it can also depend on data. If you're thinking about ember-rideshare, which again is the imaginary Ember app that it's essentially Lyft or Uber, if you have a current ride model loaded somewhere, maybe by a parent route or maybe it's some sort of service, you should be able to specify it like an async property or validation that says, "I depend on ride.state," and for all these subroutes, you would want to say that, either upon entry or any point in the future, if the state ever changes to something that I don't know how to handle that go to some default route.

That would be already, particularly in my app, which is a subset of a different kind of ember-rideshare app, that would be a huge help because the only other alternative is to build a sibling-central coordinator to the router that isn't the router but has to sort of agree with it and then, every one of these frames that you might push onto the navigation stack, they have to do some little chunk of code and then invoke this logic and be like, "Did the state change? Go where you're supposed to go," and they have to do that logic.

It would be, I think a great win for conventions as it has if it's a benefit to make people shout out their states in advance to empower them to shout out also their data constraints in advance so that you get things like automatic redirects and things change, I think that would be huge. I know that would immediately benefit off of it and I think it would fall in the same kind of problem solving that they worked on like Ember-related stuff which people don't realize how big a problem is until they see there's a better way of doing stuff. I think with that being there --

CHARLES: As an example, let's say that you're an admin and then all of a sudden, you got fired and there's an event that comes from a server that's this person is no longer an admin and it wipes out the Ember data store and then redirect you outside of the admin route or something like that.

ALEX: Yeah, that's a perfect example. To be pedantic, I think a lot of people do hard refreshes between login/sign-off stuff but if you have it all in your Ember app, that would just happen automatically. You'd still want the ability to have more graceful transitions because one of the tricky things about having stuff driven by data is that you have this giant matrix of like, "If I'm in this state and this event happens, how do I handle it? How do I make it look well-designed to the user?" But you're not going to be able to hit every one of those constraints so to just have some basic logic that's just like, "Oops, something happened," you're not an admin so we move you to the sign-in page. For in those cases, we haven't fully filled in all those leaks. I think it would be a huge win and you can just progressively decorate things according to the common flows that people take through your app.

CHARLES: You know, I'm just imagining this. Model promise, for example would be some computed property, then how would you enumerate your dependencies? Just do the mechanism that we have now? Or are you imagining something entirely new?

ALEX: I don't have a strong opinion on it because the moment I start saying what that specific syntax is, more people will agree on what's missing and what we need to have, regardless and be like, "I don't like it." I'm leaning toward something inspired by a lot of my learnings from observables, which is actually we talked about last time. The whole thing about observables is that there's almost limitless flexibility as to if you're in observable, it can take that event. It has been another observable based on that thing.

If a URL changes and you're listening to that via observable description, inside that, you could kick off another observable of Ajax request based on that URL and it doesn't make you enumerate all these things upfront. I think there is going to be a compromise between that. I think when you get into these kinds of problems, you run into stuff like Relay, which is familiar with --

CHARLES: I haven't used Relay.

ALEX: Just the idea of dynamically collecting all of your dependencies upfront before hitting the server and asking for specific chunks of data that you need, it's a very promising idea. There's cases of just dynamicism where the data comes back from the server, then you realized that you need this other piece of data and there's no way you could have collected upfront, unless you statically wrote it upfront. I expect to find that with this approach that there's going to be some stuff where you just have to be more upfront about it. But I had a cool little strike the other day on auto-computed properties and I'll also link to that. It's a different way of running computer properties where you don't have to specify your depending keys upfront but your getter function gets passed a getter function itself.

CHARLES: It's past the dependencies?

ALEX: Not even that. Imagine writing a computer property and the first [inaudible] is a function that you can call to get a property off of this but also track that you've got that property. If it ever changes, it'll invalidate again. That means if you're implementing a [inaudible] in computer property, you don't have to write first name twice, both in your dependent keys and in the actual getter in your function, which I think is kind of cool. I'm trying to make that pattern work for this data loading thing so that you don't have to have this huge verbose thing. You just lift this stuff in one place. I've sensed that the magic will probably break down in some complicated cases but that's what I'm trying to run with because I think it's pretty cool and succinct and sort of the natural evolution of what people think of as computer properties.

The other major constraint and this is also what we're talking about because it's one of the best kept secrets about the router or it's one of these things that everyone's benefiting from without realizing it, is that if a transition occurs in the router, everything in the router is going to be a possibly long asynchronous chain of operations that it collects all the data that it needs for the new routes to display. In that time, if something happens, if some hook comes along and has an exception, it can load data from the servers. If something happens then it just says 'transition.abort,' that's going to stop whatever transition is in place and you're going to stay exactly where you were and if you're not stuck in a partial transition state, that's pretty awesome. That's basically database atomic transaction semantics that people have been benefiting from if they've been using Ember for years at this point.

But again, it suffers a problem being locked away in the router. That is a cool concept. You should be able to specify like I intend this change of the state this way and if I gave you something that is logically inconsistent or can't be fulfilled, don't leave me in a weird half-assed state that I need to somehow fix and know how to fix all the different places, where I might be kicking off this transaction. I'm trying desperately to preserve those semantics when data comes into it.

One of the hardest things to do is and honestly, can be one of the hardest sells for people who are used to thinking about Ember is there's an issue of if you imagine whatever API we're talking about, it's probably going to live on the route. Some kind of hook that might be called resolve or something else, like what is the value of this context object that every function has? Is it a route? It’s tempting to want to do that and maybe, that will end up winning but winning out is the best API to get people to use.

The thing to realize is that there is no consistent value of this. This implies that there's a state of the world and you're looking at it and currently, these things have these values. But in the transaction phase, there is no stable 'this object' and you can wind up with some weird surprises. I know because, not actually these days but particularly, when a lot of the stuff landed and people started trying to do weird things and these transaction hooks, there's just like, "Why can't I grab the controller? The property isn't what I expected?" Honestly, all the stuff that is gross about query params because of this fundamental violation. You have something that pretends to be a property that is there today but is still driving this asynchronous thing that could fail.

CHARLES: I kind of viewed this as playing an off-note in the jazz thing like you only want to reserve using this, unless you're the Miles Davis of JavaScript, don't use this.

ALEX: And by Miles Davis, you just mean like the god of concurrency that's incorrect race-condition-y code.

CHARLES: Right, so it's just like you've got the right reason and you can spot the one-in-a-million case, where it's appropriate. You can spot it in an instant.

ALEX: Exactly. I'm not that person and I don't know too many people who are and that's not the API you want to land. I'm trying to, maybe wean people off on dependency on this because the way we've gotten around it in the past is to use again, is more discrete, get the value functions called 'get model' and 'get params.' These are all very in-depth stuff if you're pretty experience Ember developer but it's a way of getting a value from one of these parent routes when you're inside a transition and the rest the world can't see it but you can because you call this hook at the right time.

It's super gross because it's just a method on a route that anyone can call in any given time, whether you're inside this transaction or not. The branching logic of, "Should I look up the data from the transaction object?" because once valid, I should have get the current value of a loaded route. It's really gross to me and it causes real problems that confuse people and causes them to write issues because they've given an API that makes them feel good about treating these things as stable objects.

CHARLES: I'm trying to imagine now, just like a spike in my head. I know you don't want to get too into syntax but essentially, modeling the route tree as a set of observables, where essentially, instead of returning a promise from your model, you're just mapping an observable off of some combination of the URL state or what are the other streams of state you want to merge to realize that route. But what I'm not seeing, which I'm sure you also have the answer is the original problem, which was stack routing. What we've been talking about is making the router fully reactive like this fully reactive tree that's always on. But that problem seems almost orthogonal to the stock routing problem.

ALEX: It is. It's been very tempting to combine them. Why it is such a hard problem? Because you've got navigation stack, which almost to this route hierarchy stack that [inaudible] about but they're separate so you can't really apply the same lessons. Then you've got stack routing, which is you want the ability for routes to while they're loading, reference data that is dynamically available to them. I don't have a solid answer but I would say, the one thing that I think is going to help is that you have a few options for what you want to stash how you want to represent a URL or where you want to stash your hierarchy. Actually just track it in-memory and if you refresh the page, it'd be like, "I depend on some data that I expected to be there but it's not. It transition elsewhere," which is not a great developer experience. You could want to be able to make changes and refresh the page and continue where you left off. Otherwise, URLs aren't actually used by mobile app users.

But the other place that you could possibly put the navigation where event stack is in a query param because that can be fully dynamic and you can just sort of manage every single page. The most current page you've pop is just some top-level route but you're tracking the state on the side. I think if you solve the problem of being able to depend on things that aren't the URL or go through a more complex transition than what the router gives you by default, I think it would be possible to treat that query param or that thing you're stashing in in-memory as another source of data.

The other thing that I want to try and make sure that this new API has is really treated dependency injection where you specify all the things that you need and you don't really care from a route's perspective where they come from. I think if you had that, that would solve a lot of problems with stack routing and where it gets data from. To be very specific, today if you were in that post '1, 2, 3' comments route and you needed to access the post model from within the comments route, you would probably do this model for post.

Basically you're naming not just the model that you need. You're naming the route that you know provides it upfront, which I think is that. Actually, the real reason it's kind of the smell is that, if you ever need to change the nesting, maybe you need to introduce another level or you want to nest all that under an admin route. Then suddenly, you're asking for the wrong route name. You're not really sure all the different things you need to update if you ever change the nesting of your router. There's solutions like relative URLs that a lot of people thrown around but I think --

CHARLES: To go back in the observable world and specifically, the redux-observable world, it's like a simple map. You're just mapping down off of a global prop, you've got some tree of state and you're just mapping off... What was that like? A model hook and you're just mapping down off of that? Wherever that state lives, you're mapping to it and now you kind of slicing off your little garden hose off of the firehose. But still one huge --

ALEX: I've tried to apply observables to this problem. I don't think I've never seen the observable analogue of is this idea of dependency and injection. To model something as a stream that transforms over time, that's proven to be very useful but to sort of say, "I am an observable that expects these objects given to me," I'm not really sure what that API would look.

CHARLES: I would say, just as a straw man perhaps, you have this dependency that it's a well-known location. It's a well-known name. With dependency [inaudible] in classic, it's like, "I depend on the off service. This thing called 'service:off' or whatever. Imagine that you have some pool of state and there's some key called ‘service.off’ there and as long as I'm just basically basing my stream, the first thing I do is map off of this and maybe map off of another key and then combined those into a single stream, then I can be sure that I have those things at all times. If they change, my mapping function or my transformation function is going to get evaluated again. Does that make sense?

ALEX: Yes, I think we should [inaudible] C without code or something.

CHARLES: And maybe I'm thinking about it wrongheadedly but that would be a simple mechanism.

ALEX: Could you run by me one more time --?

CHARLES: Yeah. Let's say that we've got some authentication service that you want to depend on like you want to inject on it. You want to inject that dependency so why can't you base your stream off of that key? You have observable map, for example. The list of transformations that you would have to do to peel off multiple keys, I'm sure you could write helpers for it. But basically, probably if you're going to be wanting to inject multiple dependencies will --

ALEX: The problem is this. Basically, if you want to write your resolved observable, if this thing based on observables, remember that there is no this in a route because of the transactional reasons of what we've talked about earlier, what are you getting that from? You need to have something passed into you, to be like 'context.get observable blah.'

CHARLES: I would just assume that it's implicit. I was thinking a bit basically, the simplest case would just be an observable that was basically taken off of the entire global state or whatever of the router or what have you. The way the redux-observable works is every single epic is what they call them is just a transformation on the global stream. Usually, the first thing that happens is they map down to the local context so the --

ALEX: Like a path?

CHARLES: They have a helper like action of type, blah. You only see a subset of the actions that get maps to the Redux store. I think it's redux independent but at least in theory, every single epic is basically going off of the entire global state but the first in reality, what the first thing that happens is you're like, "I am only interested in this subset of the state," so you do a map off of the global state down to your local scope and then you work from there. In fact if you had the convention around that, you could even make that part implicit. It's like I return an observable that it's only seeing the stream of local states.

ALEX: That makes sense if there's sort of canonical state of the world but what you're doing when you're transitioning into a route is trying to feel out another state in an asynchronous manner. Redux is the action causes state to change, now the state is this. But the action for type thing, I think that makes sense if you are subscribing to the world global action on this one store when you're constructing this new tentative, may not actually become the store, you're depending on values. What we need in our API is something that depends on values that are from a tentative store.

CHARLES: It's similar so in redux-observable, you're mapping actions to actions and you're not necessarily mapping actions here. You want to get state into the equation.

ALEX: Yeah and it's so almost observables. It's just this twist of transaction dependency injection. It sounds really over-engineered but the thing is it exists in Ember today and if it exists in a less siloed way, I would certainly benefit on it. I think everyone else would too.

CHARLES: Okay. With that hand wave...

ALEX: Oh, I didn't mean for that to come as a hand wave.

CHARLES: No, no, no. I'm kidding because I think we actually have a lot more to talk about here and we're running out of time. One of the thing that I want to ask is, talking about redux-observable, talking about redux and stuff, have you given any thought as to what this might look as a library that everybody could use?

ALEX: I basically have something that's using Ember CLI only because it's so easy to just use it as a sketch pad and get test passing but everything I'm building so far is just ES6 class syntax that can be transpiled in it to whatever. I'm actually realizing, there's a lot of overlap between some of the primitives that are involved and Glimmer so it may or may not have a pass that uses references for tracking when things change until no one to invalidate and refire these async hooks. But either way, I'm going to make sure it lives in the JS usable world and not just Ember's special object model end.

CHARLES: Right. Those interfaces are pretty narrow. The things that implement those interfaces are huge and complex but the way, at least I understand it, isn't the reference interfaces themselves --

ALEX: They're really simple, yeah.

CHARLES: -- Really simple. It could almost be copied and pasted and not have much maintenance overhead in there. Here's a question and this is probably getting too far into the weeds. Can you not model a transaction as an observable? Essentially, with a flatMap, you would merge in some observable into the chain that was basically a transaction of all the other observables from which it is composed.

ALEX: You know, a transaction as it builds up all the new state over time could be part of the main tree and if there is an active transition, then that's future potential state that the world might become and it could be modeled as a leg of the Redux state. I think you could theoretically do that. Definitely worth a try. I don't think I would benefit too much from doing it now and I think this could be a premature optimization but I think there would be just quite a bit of intermediate object collection to express that. I think theoretically it works but how it's going to physically map to Ember in the near future, it would be harder [inaudible] in a way.

There's actually a lot of stuff that is very redux-y that again, a lot of Ember people don't maybe know about because it's internal but the way that Ember [inaudible], I think since Edward brought some of his learnings of Liquid Fire back to core Ember, there's this concept of outlet state, which describes -- I'm not an expert on it -- what's rendered where and then each outlet gets a chunk. Like you said, a little piece of the firehose or garden hose, pulled off the main thing so it can just focus on the one piece of state. Those are simple objects that produce this part of this transformation process. That's kind of redux-y in the way that everything just gets a new POJO and stuff changes but it's not strictly redux, obviously and probably won't become that just because it's already good enough on its own.

CHARLES: Yeah. I think it's actually good at this point to be hand wavy because the most important thing is to be non-committal about the syntax, like you said because that's when the bikeshedding begins and now it's not the phase. The phase is to come to some agreement about what is that we would love to see.

ALEX: Basically, the thing is this. I think people need to realize that Ember won the bet that the URL is an important thing to build apps around and if you have a state that's representable in URL, that state should go in the URL so you can send links around and not break the web and have an app that works that's built on half-assed routing. The only thing I'm proposing is going to make that go away. It's just that there is already this giant world of stuff that's not expressible in Ember today because it is driven by state. If you make that as easy to express and as upfront to express, I think you can have shared conventions versus what everyone is building these apps that I have to do, which is to make a sort of separate router of state-aware stuff and not have to make those two things agree with each other because it's really hard.

CHARLES: Right. At that point, you're writing your own framework. Maybe this is the next big thing because I feel like Ember usually has the best stuff way, way, way, way before. Now, we're finally getting to a point where everybody seems to realize that having a CLI is absolutely critical to the developer's experience and most frameworks aren't taken seriously until they've achieved that. It was the same thing with a router back in the day. I'm wondering what that next thing is.

ALEX: I don't know. I don't think this is going to be it. I just think it's a good progression. I think a way forward that progress is still a pretty legit central structure to build apps around and just would be welcomed.

CHARLES: When are you going to be done?

ALEX: About two or three days. I don't know. I think I'm basically going to be continuing to get feedback like the way that a lot of that original router stuff came back or it's just like constantly hit people with real examples, Ember twiddles, things are just like, "Oh, yeah. That thing. That's a cool pattern. That sucks in my app. I didn't realize that until I saw this example." These things that really teach people why this is necessary because that's going to get people's urge to be like, "Well, you could just do..." Oh, you can't because the thing that's hard to explain. It's going to be a lot of that regardless and I hope that will kick off in the next few weeks.

CHARLES: And the focus of that is going to be the ember-rideshare application.

ALEX: I think that's a good one. This is one that everyone's familiar with.

CHARLES: Have you already kind of implemented in it, like this kind of Frankenstein-ish, like this is the kind of histrionics that you have to go through in order to implement the style of routing or the style of application using today's Ember? Or have you started to begin experimentation with these new concepts and try to build out better ways of doing it?

ALEX: I'm not strictly extracting it from one app. It's sort of combined. Like I said, the few different apps that we had were an opportunity to be like, "This sporadic stuff is hard." The main route recognizer approach was an example to try different stack routing pattern. But the thing that sort of working on is drawing from three different apps and slightly different takes on it. Basically, I have something that is close to being testable in one of my main apps that will be a great chance to validate if all the stuff is as nice as I think it is going to be.

CHARLES: Okay. If the people want to get in touch with you, to help to contribute to the conversation or just publicly guilt you into moving faster towards it, how would they get in touch with you?

ALEX: I'm at @Machty on Twitter and GitHub and also, the Ember community Slack. I think I'm going to try to get people to talk about this on channel called Dev Dx Router where it's focused on development stuff all around the router. This is kind of funny because I'm talking about this thing that I've only had maybe, 12 people take a look at and comment on and begins these conversations. I think maybe some people are going to hear this and be like, "What are you talking about?" but if it gets people --

CHARLES: No, no, no. You know, the best conversations seemed to be organized around you, man. I'm just trying to think of some of the best development conversations that I've had in 2017 and you were definitely, I would say the one who fomented them. It starts with 12 people but then, if enough people take interest and be like, "Wow, yeah. Oh, man. I didn't even know that was a problem. This would be a cool way of doing it." They have a tendency to balloon and some fizzle out and some end up with real results. Anyway, I'm looking forward to it.

ALEX: I appreciate it and likewise, you're definitely one of the best people to talk about this stuff with.

CHARLES: Well, I hope other people will love listening to our conversation. With that, we'll head on out. Thank you everybody if you've made it this far. As always, you can get in touch with us at @TheFrontside on Twitter or just send an email to [email protected]. We will talk to you next week.