Part 1

Let me say, first of all, that you sure can just drop a script in your page. It's a totally valid solution.

Let's try to walk all the way to a generic solution such as Webpack (or Rollup, or Browserify or others). Notice, though, that I will be writing only about JavaScript, not other types of assets such as CSS or images or any other stuff.

So you have a script for your page. It's about… say… 500 lines of JS. Life's ok, I guess. But then you start adding more functionalities to the page and your bosses hire a couple more devs and you all work on that JavaScript and if you just continued that way, with a single script file, well… I can see that getting out of control to a big 20k LOC file. Life's not so ok any longer.

You and your team start splitting the code into different parts. You have, say, a bunch of functions to deal with DOM manipulation, some other pieces to deal with server requests, some others for data processing, a couple of components you reuse here and there, some utility functions… You quickly break the big file into… let's say about 20 smaller (~1kLOC) files. And how do you manage these? Well, you just insert 20 <script> tags in your page, in a carefully chosen order, because, well, you have to be sure that that utility function is available before you try to use it, right?

Now life is… well, better, yes, but it's not that nice either. Because you have to keep track of where you put your functions, the order in which things are loaded and all that. And files, well, 1kLOC files are still a bit big, you know. But if you split more finely, you'll get many more files and that means more <script> tags and more loading issues and… Ok, life is not really too good, I guess.

Not only that, but you know, all those files define a lot of names. Names of functions, names of variables. And those that get thrown in the global scope… uh, yes, they may collide with other names defined in other files. So adding new stuff gets complicated.

So you reach out to your knowledge (or consult some book, tutorial, ask for help, whatever) and you discover the “Revealing Module Pattern”. I won't explain it in detail. Suffice to say that it is a structure like so…

    let something = (function() {
        // "so called private" code here
        // ...

        // and then...
        return {
            publicOne: ...,
            publicTwo: ...

…which basically provides you, through a closure, with some encapsulation. The thing returned and assigned to something has some visible methods and/or properties, and those methods have access to the local stuff defined inside the function expression which no one else has access to. So, to some extent, it is a structure that allows you to write some encapsulated blocks with “private” visibility. Why this is good for your problem? It avoids name collisions. What you do is each of those 20 or now 50 files you have, you create this structure, engulfing all the content of the file. And at the end you only return the things you really want to be visible.

This is a huge gain, because you can now split the files as small as you want without concern for name collisions. On the other hand you now have 50 <script> tags or a hundred. And that thing with the order, well, life's not so nice in that front.

Let's recap a bit:

  1. You've successfully solved the problem of having a single huge file. It was a problem, because it was really huge and because you needed to have various people working on it at the same time and that's nasty. But now that's solved because you have separate files and one person can work in some files while another works in others.
  2. But you now have these problems:
    1. You have 200 <script> tags in your page.
    2. They need to be kept in a certain order.

So we're going to try to solve these two new problems.

There's a really simple solution. It won't solve everything, but it is simple: You could have a shell script or some similar tool that simply concatenates all your script files in the correct order. That way, the first problem is clearly solved. Your sources are in 300 files, but the script included in the page is back to being just this one file. So just one <script> tag again. Great!

But the second problem remains. For the shell script to correctly concatenate the JS files, you have to tell it what's the correct order. You can go through many naïve solutions here. Some may even work to some extent. Say you named your files following a certain pattern like 00100-somefile.js, 001300-anotherfile.js… and then just concatenate following the number. It sort of works. Not pretty at all to maintain, but it sort of works. Or you could keep somewhere in the shell script the ordered list of files. Or…

Anyway, any solution along those lines is still a kludge and doesn't really solve the problem. So instead, you think of a pattern a bit more sophisticated. To the above pattern you try to add some way for a particular file to say what other files it needs to be available before it can run… Its dependencies if you will.

I don't want to write what that could look like in a naïve approach, but you can look into RequireJS for an approach that is somewhat close to what it could look like.

Then again, while you're doing all this, some folks publish NodeJS, it becomes popular and as it precisely includes a mechanism to do exactly this (defining “modules” which can export some public parts and require other modules) then that mechanism, with that particular syntax and all, becomes very popular too. Note, that later, the standard decides on a different syntax and mechanism but that doesn't really matter much; the important bit is that there are some particular syntaxes that become popular, and so you follow that.

But of course, the syntax in NodeJS works well for NodeJS. And while ECMAScript finally standardizes on another syntax, the point is that these systems are designed towards really having multiple independent files… and you don't want to serve your files separately.

The syntax leaves out most of the “weird” boilerplate about closures and simply allows you to write it in a manner such as…

    // one of these to require dependencies:
    let a = require('a.js');
    import a from 'a.js';

    // ...your code...

    // One of these (or other similar variations) to make things visible out of your module:
    module.exports = something;
    export something;

So you revise that shell script you had that simply concatenated all your JS files, and turn it into something a bit more sophisticated.

On the one hand, you make it so that before and after each file's contents a little bit of boilerplate is added. This is, approximately, the same closure boilerplate you had removed in exchange for this new syntax. Not exactly the same, but it is similar. The main difference is that it is putting your code inside a function expression but instead of simply calling it directly, it passes it to a small function that the shell script makes sure is included first thing into your final “bundled” JS file. That function _will_ execute the function that it receives, but it will do so with a bit more control. Because it either manages those require(“a.js”) calls or transforms those import from “a.js” into calls it can manage. And it will capture the returned output from the module (i.e. the parts you decide to export) and it will manage those too. In what way? Well, it keeps a registry of modules. That way, the system knows what it has to give your module when it asks for a.js and also what it has to give other modules when they ask for yours.

Not only that, but having such a registry, that keeps track of what is loaded and available also solves the order problem. If it is in the registry, it has already been loaded. If not, the system can delay your requirement or your importing until that file is indeed loaded.

Let's recap again.

  1. You have a manageable code base, where the sources are split functionally into small modules. This is a very good thing.
  2. You have a process or tool that:
    1. Puts all the small files into one big file
    2. Adds some completely generic utility functions that:
      1. Take care of providing each module with any other modules it asks for
      2. Solves the problem of order.

Note that it's not only this. Really the dependency system can work with your code, whether it is encapsulated and bundled into one single file or whether it is left as separate files and loaded on demand when they are required or imported. As long as the system or the tool provide the mechanisms and understand the same syntax you gain that ability for your code.

Now, this tool is something you can do yourself. But it would be much better it lots of people used similar tools and if those tools worked in the same way. That way, you could treat external libraries in the same way you treat your own code. So, instead of actually building yourself such a tool, you use an existing one. One of those tools is Webpack, though there are others.

A final notice: Some of these tools, as others have mentioned, tend to take advantage of the fact that you are doing all those code transformations and bundling to offer doing other tasks too. Such as minifying your code. You know, compressing it so that it's smaller and is loaded faster. Or maybe they can even avoid including files which are not actually needed. Or the parts of files which are not needed. Or they can also process other assets such as CSS and/or images. Once you've agreed to have that tool or process as step in your workflow, well, why not make the most of it?

So this, in a generic way, with many inaccuracies and over-simplifications, is what these tools (not only Webpack) are about.

Part 2

Thanks everyone for your kind responses. Ah… I don't know if I'd post this in a blog, but feel free anyone to post in your own if you so desire. You don't even need to give credit to some random letters and numbers on reddit xD

But anyway, I had to stop but forgot to add a bit about how it really happens. I'll use Browserify because its simpler. Don't mind much the code itself, it's not a complete example. (Also, the following is not needed to just get a general idea, but it might help in understanding how it is done)

Say one of my files (something called linkLoader.js) looks sort of like this:

    const xhr = require('../lib/xhr.js');
    const dom = require('../lib/domUtils.js');

    function loader(container) {
        const output = dom.printTo(container);

        xhr.get(href, function(content) {
            var { content, js } = dom.parse(content);
            // ...

    module.exports = loader;

I've removed most of the code. The interesting bits are still there. So, you run browserify on this file and it spits out a bundled file. I won't show all the result of that here because it's too big and noisy. But, this particular part gets transformed into something like:

        1: ...,
                    const xhr = require('../lib/xhr.js');
                    const dom = require('../lib/domUtils.js');

                    function loader(container) {
                        const output = dom.printTo(container);

                        xhr.get(href, function(content) {
                            var { content, js } = dom.parse(content);
                        // ...
                    module.exports = loader;
        3: ...

So it gets thrown into an object. This object will be passed to the function I mentioned that will execute each of those. As you can see, the transformation is mainly just wrapping the original code and extracting the dependencies that each module requires, for easier management later.

It is interesting to note that your code then gets executed in an environment where you have access to three things: a require function, a module and exports references. This is basically all you need for your code to work and it is interesting that it doesn't really matter much what these are or how they work at a detailed level. Just that they do what you expect. This is what allows what I mentioned earlier: the actual loading can happen like it is done here, in a single bundled file or it could happen in some other way (e.g. by loading it on demand through XHR or from the filesystem or whatever).

If you want to actually see what those things look like or what the general function at the start of the bundle looks like, you can have a look at the browser-pack package. But a general idea might be doing something like this.

    I have that object there with all those functions so
    forEach(key, module) -> {
        funct = module[0]; dependencies = module[1];
        registry[key] = execute(funct, getDeps(dependencies, registry) );

This, of course is a very naïve approach. A real solution needs to take into account the availability of dependencies before they are used. It also doesn't really work like this at all in regards to that registry because your code does not return. Instead you add things to the passed module.exports or exports, but that's just a detail.

Now, I've used Browserify because it is much simpler than Webpack. The output Webpack generates is similar in spirit. Webpack builds an array, instead of an object and wraps our modules into something like this:

    /* 1 */
    (function(module, exports, __webpack_require__) {
                    const xhr = __webpack_require__(0);
                    const dom = __webpack_require__(3);
                    // ...

(The comments are there just for human debugging purposes.) As you can see, the main difference is that Webpack applies some transformations to your code while it is generating the bundle. The main transformation is that one you see with __webpack_require__. Not only it does change the name or require to that, which is a superficial change, while it is doing that, it also removes the reference to actual filenames and substitutes them for a simpler index number. In any case, the result is similar: All the benefits explained in the previous message are there.

Also, as already mentioned, Webpack does more than this. This is all in relation to modules. But Webpack also includes other tasks which you might do with other software. Like compressing (minifying) the output file, or managing CSS alongside JS, or running a transpiler… Or a common one. As I mentioned there are mainly 2 different syntaxes for importing and exporting. The CommonJS (what NodeJS uses) and ESM (the ECMAScript standard), i.e. require('bla.js') vs import from “bla.js”. While Browserify only supports CommonJS, Webpack supports both by transforming “at pack time” those imports into requires. (Note that this isn't strictly correct. Webpack 1 didn't support import either, but Webpack 2 (and 3) does. And also, you can get the same result combining Browserify with other tools -Babel- so that they do the transformation and then Browserify does the packing.)

Now, there is just one remaining thing you may be wondering about. It could be something like: “Well, now that there is a standard way to load modules, can't we just use that and forget all this about bundling it all into one file and just let the browsers load what they need?”

The answer to that is not completely straightforward. Let's just say…

  • While there is a standard (mostly, some details still being heavily discussed), there has been no available implementation of it in any browser until… well, very recently. The very latest versions of some browsers are just now starting to ship with (some) support for ES modules. (See the warning at the top here).
  • So in the future it may be a way or the way, but for now, needing to support current browsers, the solution does seem to inevitably go through a bundled file or some similar solution that offers the functionality browsers don't.
  • There are also some other things that affect usage of all this. In particular, performance concerns and HTTP2 support may help or may not help going back to multiple independent files being loaded. This is a bit hard to determine yet, but it may mean that in some scenarios bundling all files into one (or a few) file(s) might still perform better.

So the answer to that is a classic it depends. Or, if you prefer, it could be something like: “For now, bundling is a good idea in many cases. In time, we'll see”.