Concurrent Programming: Linux Memory

While I finish the blogger version, you can have a look at the original entry in my GitHub account.

TL;DR:

One practical take home message is this:
When using top in Linux, it's not meaningfull to add the RES value of different processes as that does not amount to the actual, combined physical memory used by all of them. It's also not meaningful to add the values of USED for different processes; neither is it to add the values of RSsh: They do not account for independent chunks of physical memory; we are overcounting in general!
See the examples in that post.

What Is Science?

 We all "know" what Science is...at least at we have a pedestrian working definition of sorts: based on experimentation, models and theories need to explain known facts but also predict new ones that then must be confirmed...by experiments. Some say this as they must be refutable -as in the opposite of not even wrong.

Here I want to suggest another, hopefully equivalent, idea. Something that has bothered me for some time now is that naive realism that I feel lurks under expressions like the following attributed to Max Planck

An experiment is a question which science poses to Nature and a  measurement is the recording of Nature's answer.

The following ideas are closer to a different quote attributed to Carl Sagan

Science is a way of thinking much more than a body of knowledge

So, What is Science then?

Looking through the window you might say "the trees are tall, the grass is green, the hill is smooth, the dairies are white and yellow,...".

That is poetry, literature. It's another approach to reality, to learn about ourselves in relation to others and to Nature. And it is a necessary one.

But it is not Science. Nor those are the only type of sentences we can come up with.

Here some different statements: 

• "When the sun shines flowers bloom, grass grows and becomes greener, trees get more & greener leaves,..."
• "When humans constantly tread on the grass, the grass dies and the area becomes sandy & arid"

Wait, "dies"?? What's that? What is death? A possible answer would convey the following idea:

• "When we provide food to a living being it grows and multiplies. Eventually it stops doing this and we say then that it died."

All of Science is in essence a collection of sentences of the type

When we do/it does this, it happens that

In order to understand the it, we (let) apply "transformations" on it -the object of the transformation-, and "collect" the output -which itself is an it! 

I saw a parallel with Category Theory: Objects are secondary; what matters are the morphisms. These tell us everything about the objects.

In Science, experiments are but transformations.

Stated like that, it reminds me of the warning of Carnap about Science: it's a combination of experiment plus theory. Theory as a mental construct/model.

Q1: What is theory in our view?

Ans: Our view doesn't change this, but it may give it a spin. Theory & models are but a mathematical guide for us to do

• Actual experiments
• Gedanken experiments

And thus anticipate the outcome (or rather, how the outcome behaves). 

As these mathematical constructs can themselves eventually be expressed within CT, we could justify the claim 

Science is about applying & combining transformations

The inputs & outputs are themselves understood as transformations, thus it seems redundant to include them in the above claim.

Q2: This still seems to miss Carnap's observation that Science is not a collection of facts. A better formulation?

Ans: Science is about applying & combining transformations...and the talk about it.

We mentioned transformations as synonyms for actual experiments.

The talk about it is our on-paper, mathematical experimenting plus the verbal talk and descriptions. The verbal talk can eventually be fitted within CT.

Q3: Usually one stresses the fact that scientific method must be 1) predictive and 2) refutable. Are we missing this?

Maybe we can introduce a "why":

We experiment and talk about it in order to be able to anticipate (predict).

Consider the following:

"If you drink this potion and repeat 10 times these words, the gods will vest you with the power of invulnerability against bullets"

It talks about transformations (drink potion, utter words), has a prediction (invulnerability against bullets) and is refutable.

Our goal to anticipate makes us try ti make sure it works -instead of just take for granted-, and thuscto refute it. And it turns out to be false.

Maybe then the "why" and its answer "to anticipate" comprise those common two traits.

We come then to the following statement:

Science is about applying & combining transformations and our talk about it in order for us to anticipate (the outcome)

What I find so much appealing about Category Theory is that it applies the same approach to study mathematics itself. Instead of mathematical object&structures it shifts the focus to the morphisms (transformations) on & between them plus the properties of combinations between those morphisms.

For example, and this is a general modus operandi, not something CT, if we want to understand different types of graphs we could run a set of stochastic processes on them. One same stochastic process may proceed differently on different graphs. Ultimately this allows us to distinguish and classify those graphs.

That "talk about it" is thus a talk  in terms of applying and combining mathematical transformations. 

Ingredients for a Successful DevOps Practice

...At least three basic ingredients.

If I would have to pick what I think could be three basic ingredients for becoming a successful and valued DevOps, it would be the following.

First and foremost one has to have a passion for automating things. It's the "hate to do things twice" attitude. But this is not software development per se. The goal here is to abstract away a process in a manner that gives us

1. the possibility to control, test and run it with a range of options and
2. to grant us a meaningful birds-eye view of that same process that will facilitate the discussion of it and of our infrastructure as well as eventually lead to further improvements of it.
   

It is automation as a way to gain a better overview of the infrastructure, its uses, and the needs for the business

Second ingredient, the "how you achieve automation of a process". Make a real story of your Jira story! Don't shy away nor be lazy to add comments full of details. This is the place where you don't need to worry much about formatting, yet you have all the tools to insert screenshots, links to other boards or documents, link it to further stories, tasks or even epics, explicitly raise the attention to other people that may or not be in your team, add snippets or even full scripts code, etc. Most importantly, it's a single stop shop where you can

1. First just jot down an idea of a project
2. Draft the requirements and goals
3. Annotate some initial tests and investigations for solutions
4. Divide the story into tasks and littel milestones
5. Go to step 2 and keep improving until a satisfactory solution is found
   

Again, you get to do all that without worrying about the eye-candy of formatting not even constantly writing full sentences.

Clearly, the story is a living object that may evolve in content as you advance in its solution. And the good news is that it is always there.  In the short term this helps gain accountability as well as keep in check our logic. In the mid and long term this attitude serves the traceability of the decisions and solutions taken. Others will certainly benefit greatly from it!

The third and last basic ingredient I would highlight: Show due reverence for the logs. Any scientific field, be it Physics, Math, Engineering or even Language or Humanities can be understood as a composition of processes. A network of maps each with its inputs and outputs: the output of each process is the input of the next.

The logs of your automation include any of its required outputs (say updating a bucket, building a stack, triggering a pipeline, etc.). But it's vital that its output be seen as a story in itself: Reading the logs of your scripts should immediately provide us with a clear view of a

1. Full path of your script as it was called
2. All values of all variables used
3. timestamp of each step and two cents of what it's doing
4. which function in the script is doing what at that time
5. which function ended in error and in which line of the script

And that should be done with a formatting that is

• consistent
• easy to parse

I can hear some of you already shouting "how does this apply to Terraforming or CD/CI?". 

Well, you better structure your Terraform code into modules -yes, beyond the basic main.tf, variables.tf and terraform.tfvars. You see, It isn't then any different! And the whole CD/CI...There is a reason why the mathematicl concept of a map (for CS minded readers, think of lambda calculus, or a language like Lisp or Ocaml) is so powerful: It conveys the most basic idea of a path/process, among a plethora of other technical details. And that's what our pipelines are.

Of course is easy to keep adding ingredients: Eagerness to constantly learn and use new technology, embracing the KISS rule, be quick in documenting things, ... But if I'd have to pick a fourth one I would first go with a 

Trust the adage "good is better than perfect".
 

Although I first heard it as "Perfect is the enemy of good". You get the idea though.

Tutoring

 Tutoring

I asked about an update on how a former student finished their course last June.

"X's math course did not go well: he ended with a 70. Way below his ability. Not sure what happened but we are hoping that the return to in person learning will help him regain his footing academically. "

It's not satisfying. 

I'm definitely sorry to hear X couldn't keep that higher mark that he was getting during the course.

The course structure didn't help. Yet, my task was to at least keep him at that higher mark range. So, that's on me -at least in part and to whatever extent I can influence that.

 

From the point of view of the student, every tutoring puts a little burden on them to try to match the explanations from the teacher and the tutor relative to the same concept. 

From the point of view of the tutor, the task is in part to find the adequate strategy to deal with the usual first questions the student will raise: "I don't understand it", "I don't know how to do it" or "I don't know what I'm supposed to do". 

 

More often than not, the student knows it. The problem instead lies on making the connection between a problem's statement and their knowledge of the subject. The reason may range between two ends. 

 

On one end the student knows the mechanical steps, "the how", but doesn't quite understand, and even feels at lost with, why those steps are as they are or what they mean. This makes it hard to solve open-ended problems, where inquiry and thinking are the major skills leading to a solution. Often the latter is simple in terms of calculations required. The hard thing is to come up with the idea of calculating it that way.

 

On the other end, the student has some (a reasonable?) understanding of the general idea, but lacks in enough concrete  practice  (i.e., hasn't practiced enough, hasn't done enough exercises yet). Two examples may help: Adding fractions will always show up in math, after they see it in grade 6/7. A weak background on it will become a frustrating drugging force later on when facing Math in grades 10 and beyond. A different case that always shows up is at the beginning of a new topic. They may have worked out some examples in class, but the concepts, while making sense when explained, are still not completely clear in their minds, and even less so are the calculations involved.

 

I see Y as falling closer to the first case, while X, however, would lie not far from there, but somewhat more towards the middle between both ends.

 

I'm thinking that in cases like X special care should be taken. Finding the right balance between working on the concepts and working out cases is tricky.

Now I think for X it would be best to keep much more concrete on the task at hand, the exercise to solve. Furthermore, we should deviate as little as possible from the procedure followed in class by their teachers.

Despite all effort one may make, sometimes the differences between the teacher/tutor's approach and the needs of the student can't be brought enough in sync.  Changing the teacher/tutor may then do the trick.

 

It may well be that oils boils down to me emphasizing more problem solving that concept discussion.

Following the motto of "Be pragmatic!"...as that's the simplest, and most common measure of what one has worked on with the students. 

 

Feeling a bit frustrated. Let's see what the parents reply and if I can get more feedback on what happened...

 

Number 1

I had other topics pending that I expected to talk about before this, and also earlier. But these months my new job got my attention away from the blog. I may comment on this in a future post -or not.

So, what brought me back? Well a problem that I've pondering over since about 4 years ago when teaching Computer Science at a High School level (Grades 9 to 12). The problem shows up in CS very early: What does repeatedly right-shifting a binary number represent? In essence, the (integer) logarithm in base 2, yes. But, division is a concept well controlled by any grade 9 student. Why not then introduce the logarithm already then?

We would need to 1) show how the laws of (integer) logarithms can be justified from repeated division, 2) show how can we calculate the general logarithm by only using integer logarithms, and, finally, 3) show the geometric description of the logarithm. These days I realized how 2) looks like. Hence this post.

Initially, I thought it could be more or less straightforward to derive the usual properties of the logarithm from those of the integer logarithm. Also, the idea of visualizing the logarithm and exponentiation as paths towards/away from 1 was appealing. After trying building the logarithm from its integer version, that seems now misguided. It is straightforward to introduce and justify (almost) all of the properties using the integer logarithm for powers of the same base. But in general that won't work: $ilog_25=2,\;ilog_35=1,\;ilog_23=1$ and clearly $ilog_25\neq ilog_35\, ilog_23$. However, the visualization as paths is straightforward.

I already wrote some time ago a few posts on the topic elsewhere. As I said back then, I like the number 1. It seems a dull choice and many may go for other more mysterious numbers like 4, 8 or 24. But there are some key elementary concepts that hinge on 1.

One of these concepts is the basic operations of multiplication and of division. Another (related) pair of concepts are the Logarithm and Exponentiation. The basic idea may be stated as what are the “paths” that lead to 1.

Geometry of Division and multiplication

Let's first view this last statement graphically.

Fig.1 Geometric construction of Multiplication and Division

 

 Point $U$ defines the unit of measure on axis $Px$, while $U'$ is its counterpart on axis $Py$. Then multiplying $A$ times $C$ yields point $AxC$, while the quotient $B/C$ is given by  the point $BoC$. It is implicitly understood that all measures are in units of $OU$.

Hence, for $0<a<1,\;b>c>1$, both, repeated multiplication of $a$ times $c$, as well as repeated division of $b$ by $c$ can each be seen as a path towards $1$.

The same construction rules show, that $\lim_{n\to\infty}a^n=0,\;\lim_{n\to\infty}c^n=\infty$. Thus exponentiation corresponds to paths that go away from $1$. What operation do the paths of repeated divisions/multiplications without crossing $1$ correspond to ?

 

Note: This geometric construction of multiplication and division is valid in a projective space. Indeed, while the previous figure seems to show the constructions paths parallel (in an Euclidean sense) to either $UU'$ or $CU'$, the whole construction has been made as a projective construction with the line of infinity far enough.

The Logarithm

The paths following repeated multiplication or divisions without crossing the unit lead to what we call the logarithm. Already from a geometric point of view, the logarithm seems "opposed/inverse" to exponentiation. 

The Integer Logarithm $ilog$

How many times can we divide $b$ by $c$ without crossing the unit $U$? or How many times does the "division-path" bounces off the $Py$ axis? Ans: $ilog_cb$

How many times can we multiply $a$ times $c$ without crossing the unit $U$? or How many times does the "multiplication-path" bounces off the $Py$ axis? Ans: $ilog_ca$

We will convene that when talking about multiplication paths, we will give the answer a negative sign. Hence, for the values of Fig.1, $ilog_ca=-1$ while $ilog_cb=2$: dividing the $BoCC$ further by $c$ would lead to a values smaller than the unit.

General Properties of the Integer Logarithm

From the definition the following properties follow:

0.- $ilog_ca<0$ when $c>1,\;a<1$

1.- $ilog_c1=0$  If we divide/multiply the unit by $c$, we immediately move away from it. So, no paths available.

2.- $ilog_1a$ is undefined as we can both, multiply or divide $a$ by $1$ infinitely many times giving thus both $-\infty$ and $\infty$. In $\mathbf{P}\mathbf{R}$ we could evtl say it's the point at infinity.

3.- If $c <0$ or $a<0$ then $ilog_ba$ is undefined. It's either impossible to lead a path towards $1$ (when $a<0<b$) or totally meaningless ($b<0$).

The following are not clear-cut derivable from the definition, but seem quite plausible:

4.- $ilog_b m\,n =  ilog_b m\,+\, ilog_bn$. Let's see why: Be $k=n\,m,\, i= iLog_bn,\;  j = ilog_b m$, all integers. For simplicity sake, we’ll assume that the logarithms are exact. $ilog_b k$ corresponds to dividing the rectangle area by $b$. The side $n$ can be cut by $b$ $i$ times. That still leaves a rectangle of area $m$. This can be divided by $b$ $j$ times before the resulting area gets below $1$. Hence the original area can be divided $i+j$ times by $b$. qed.

5.- $ilog_ba\,=\,1/ilog_ab$

bla, bla, bla (I'll come back to this later. Right now, just wanted to get all this off my head)

Calculating the Logarithm using only Integer logarithms

Let's first state the result:

Be $a, b \in \cal{ R}^+, \; 1<b<a$, then
$$\log_b a = l_0\, +\, \frac{1} {l_2\, +\, \frac{1} { l_4\, +\, \frac{1} { l_6\, +\, \frac{1} {\cdots}}}}$$
Where $l_0\, \equiv\, i\log_b a$ is the integer logarithm of $a$ in base $b$ and $l_{2i}$ are all integer logarithms as well.

We will now clarify and justify this expression as a continuous fraction.

Be $b_0\equiv b$ and $b_{-1}\equiv a$. Then
$$
\frac{b_{i-1}}{b_{i}^{l_{2i}\,+\,l_{2i+1}}}=1 \\ \quad
b_{i+1}=\frac{b_{i-1}}{b_{i}^{l_{2i}}}$$
with $l_{2i} \in\mathbf{N}^*$, $0<l_{2i+1}<1$ and $1<b_i<b_{i-1}$.

From the definitions of logarithm and integer logarithm there are $l_1$ and $b_1$ such that
$$\frac{a}{b^{l_0+l_1} }=1\,, \;0<l_1<1 \\
\quad b_1\, \equiv\, \frac{a} {b^{l_0} }<b\\
\frac{b_1} {b_0^{l_1} } =1$$

To calculate $l_1\equiv\log_{b_0}b_1\,=\,\frac{1}{\log_{b_1}b_0}$ and we proceed the same way as before. Hence $\log_{b_1}b_0 \,=\,l_2\,+\,l_3$, where $0<l_3<1$ and $l_2=i\log_{b_1}b_0$.

 In this last figure, point $A$ lies right off the limits of the picture. The red dot-dashed line shows repeated division by $B$, without crossing the unit $U$. This leads to $b_1$. Now do the swapping $A\gets b$ and $b\gets b_1$ and repeat. This leads to $b_2$. Each time count the number of bounces off the $Py$ axis (the one containing $U'$): That's gonna be $l_{2i}$. In this case, therefore, $l_0=3,\;l_2=2$.

Either some $b_i$ hits the unit and the process ends, or it never stops. In this first case, the result of $\log_ba$ is a rational number; in the second, an irrational (and transcendental) number.

In short, using just the operations of  (repeated) multiplication and (repeated) division, given two numbers, $b,\, x$,  the logarithm Log_b(x) is defined by paths leading towards $1$, while the exponentiation $b^x$, by paths leading away from $1$.

What can I say. I like "seeing" things and this geometric view is something fun.

Further Geometric Intuition 

I think there is way more intuition to uncover from a geometric view of the logarithm. I hope to describe some of it another time soon. For now, let me collect here some plots I had done and posted elsewhere before.

Note added in proof

Of course, I had to find only now, that this is Daniel Shanks' algorithm for the logarithm published back in ... 1954 (https://www.jstor.org/stable/2001992 or https://www.ams.org/journals/mcom/1954-08-046/S0025-5718-1954-0061464-9/S0025-5718-1954-0061464-9.pdf) !

I still think that explicitly referring to the integer logarithm, as repeated division, adds a clear pedagogical value to it. Further so, if the geometric construction is shown.

Furthermore, for all pairs $0<a,b$ the logarithm $\log_ba$ can be thought as the continuous, projective "path" (as repeated rescaling) from $a$ to $1$ parameterized by $b$.

This is more of a wish list, or reminder of a vague idea, that would require to be made more precise.

On Google you can find a video from Mathematica showcasing the continuous fraction expression for a logarithm. See also this tweet or this site. 

I found about Shanks on https://exstrom.com/blog/abrazolica/posts/logarithm.html

Some ideas for when I need a distraction

Only on an Euclidean setting we get the usual values of $log$ as obatin by any calculator. What does the same continuous fraction corresponds to in a Projective space?  E.g. $\mathbb{RP}^2$ or $\;\mathbb{S}^2$

Likely more interesting would be the case of $\mathbb{HP}$ or $\mathbb{HP}^2$, where we loose the commutativity but still retain associativity, and thus, Desargues theorem, which is the only one needed for this construction to work.

20 Year Old Bug and Counting

That selfishness character in some open-source projects

 Continuing with the topic of web servers and my work, as of the last days, of writing up some hands-on examples, I'm writing this post about a bug I found in Firefox. It's not new. 

 Before I do so, I do not want to pass the opportunity to touch on a topic that has been bugging me for quite some time now.

 Let me start with a few examples.

I downloaded the source-code of Firefox -read below why. Nothing on the Mozilla project page alerts you of the size of the download required. It turns out that code is about 4GiB of download. But then you have to run some configuration scripts which also download some more stuff.

Don't ask me how much though: it's not mentioned nor notified anywhere!

That download forced me to update my Homebrew installation on my laptop. Yeah, I hadn't done so in quite a while. Hence, there where plenty of packages that were updated. Well, not always do you get warned about how much data it's gonna cost you nor will it ask you for confirmation to proceed.

While writing my previous post I decided to test the download of River, the Apache project  originally developed by Sun under the name of Jini: Nowhere does it say how large is it gonna be such download...although this time a gzed-tarball ended up being somewhat less than 6MiB.

While browsing Bugzilla about the aforementioned bug, I came up way to often with a reply by what seemed Mozilla developers asking the posters commenting on buggy behaviour for a minimal example that would showcase the problem. It so is the case that what brought me to find out that same bug was precisely writing a minimal-case scenario for use with Firefox, namely, a basic server delivering some very simple HTML code back to the client.

So, developers of a huge project like Firefox, answering a question related to networking...and they cannot write themselves a minimal server in C that serves some 10 lines of HTML?

 When I was doing Sys Admin work and would receive some request about an incident or something not working as they expected, I would always 

1. Quickly try to guess what might really be going on, then
2. Come up with a minimal example showcasing what my guess was, and only then
3. Ask the user for more clarification, if my little experiment wouldn't replicate the problem.

I find both types of examples unhelpful and selfish -for the lack of a better word right now. That's all I'm gonna say about that and open-source right now.

 

A 20 year old bug

And now about what I promised.  Here is the bug:

Any web page whose HTML code starts like the following will trigger the bug in Firefox

<!doctype html>

<html>

<head>

<title>A 20yr-old bug</title>

<meta charset='utf-8'>

</head> 

<body>...</body>

</html>

The bug consist in Firefox duplicating the GET request: The full pages loads twice!!

While this seems irrelevant, there are case scenarios where this can become an actual burden. 

 

Some mobile plans do not offer that much of data, and what they offer is expensive. 

 

Among those plans are Roaming. The following roaming plan is the first that popped up when googling (prices are in CAD):

 

| Area \ Co. | Rogers  ($/MiB) |

| US            |           0.16           |

| EU            |           0.50           |

| Asia          |           1.00           |

| South A.   |           1.00           |

Mind you, a typical newspaper front page, e.g., the one from The Guardian below amounts to somewhat over 1MiB of data -there is not just text, but also pictures & ads.

So, If you went to visit your family in Malaysia and check out a newspaper with such an "stutter"-triggering HTML code, you'd be paying $\$$2 per page view if you use Firefox! Double as much as if you were using Safari, $\$1$. I think Chrome also works well -haven't checked yet-, hence only $\$$1 as well.

 

Why?

Because, not knowing better, those at Mozilla decided once that it was easier for the developers to force the browser requesting again the page every time the charset meta tag is not the first tag within the head tag!

So, this HTML code doesn't trigger the bug:

<!doctype html>

<html>

<head>

<meta charset='utf-8'>

<title>A 20yr-old bug</title>

</head> 

<body>...</body>

</html>

 Here the main bugzilla entries related to this tag. Notice how along these over 20 years the bug has been repeatedly rediscovered, fixed or simple disregarded!

1. Bugzilla entry opened 20yrs ago updated 10 months ago. That is, it's still open -as of October 2020!
2. Opened 17yrs ago, closed 8yrs ago...Status: Resolved!

Various postings in both of them show people have been re-discovering it and landing on either bug entry depending on their luck.

 

How often does that happen?

I don't know, yet.

 

But I do know that the newspaper The Guardian will trigger that bug

 

The same applies to The New York Times

or the french one  Le Figaro

 

Newspapers like Toronto Star,  Zeit Online, or El Diario.es are "stutter" free: they have a charset meta tag that come as the first tag within the head tag.

All this applies only to their front pages, which is the only HTML code I checked.

 

Google search page in Canada doesn't seem to have any charset tag, so it won't trigger this bug.

Ah, that's funny what I just saw: Using Safari at least, as the pics above show, Apple's web page www.apple.com does not show any source code! Very reassuring, Apple... Ok, checked with Firefox's Inspector  (under tools->web developer menu) and, indeed, apple has a charset meta tag deep down its head...

I need to double check this bug while capturing packets with a tcpdump... This duplication of requests could be happening way more often than I initially thought every time we use Firefox to browse the web!

 

I'll update this post accordingly. Stay tuned.

A minimal setup to trigger the bug

You can do it just in a high-level, meaning setting up a webserver with a counter of how many users have requested a given page and showing that number to any one who connects.

Another way is compiling the following C code and running it. If you do so in your local machine, point your browser to http://localhost:8080. Do it using Firefox and other browser like Safari, Chrome, Opera, Konqueror...

 

Whatch how the counter jumps always by 2 when using Firefox!

 

#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <string.h>     //memset
#include <unistd.h>        //close

#include <stdio.h>        //printf
#include <stdlib.h>        //calloc

#define PORT "8080"
#define RDBUFLEN 1024
#define WRBUFLEN RDBUFLEN

int main(){
    struct addrinfo hint;
    memset(&hint,0,sizeof(hint) );
    hint.ai_family = AF_INET6 ;
    hint.ai_socktype = SOCK_STREAM ;
    hint.ai_flags = AI_PASSIVE ; //for use w/ bind

    //getaddrinfo
    struct addrinfo *sadd ;
    getaddrinfo(0, PORT, &hint, &sadd);

    //socket
    int server_s = socket(sadd->ai_family,
                sadd->ai_socktype,
                sadd->ai_protocol
    );

    //bind
    bind(server_s, sadd->ai_addr, sadd->ai_addrlen);

    //listen
    listen(server_s,3);

    //accept: blocking : waits for client to connect
    struct sockaddr cadd;
    socklen_t cadd_len = sizeof(cadd);
    int client_s = 1;
    int n=0;
    while( (client_s=accept(server_s, &cadd, &cadd_len) )>0 ){

        //recv
        char*  buf = (char*) calloc(RDBUFLEN,sizeof(char));

        size_t rcvd = recv(client_s, buf, RDBUFLEN,0);
        *(buf+rcvd)='\0';
        printf("Received:\n%s\n",buf);

        //send
        char response[WRBUFLEN] =
            "HTTP/1.1 200 OK\r\n"
            "Connection: close\r\n"
            "Content-type: text/html\r\n\r\n";
        send(client_s,response,WRBUFLEN,0);

        sprintf(response,
            "<!doctype html>\r\n"
            //"<html><head><meta charset='utf-8'><title>Lemon Inc.</title></head>\r\n"      //Firefox sends 1 request only
            //"<html><head><title>Lemon Inc.</title></head>\r\n"                          //Firefox sends 1 request only
            "<html><head><title>Lemon Inc.</title><meta charset='utf-8'></head>\r\n"  //Buggy Firefox sends 2 requests!
            "<body style='background-color:#000000;color:#fff'>\r\n"
            "<h1 style='font-size:48;color:#ffff33'>Lemon Inc.</h1>\r\n"
            "<p>Your browser sent me following details:</p>\r\n"
            "<div style='text-align:left;width:50%%;margin:auto;'>\r\n"
        );
        send(client_s,response,strlen(response),0);

        int sent=send(client_s, buf , rcvd,0);

        sprintf(response,"</div><br>Number of connections today: %d\r\n",++n);
        send(client_s, response , strlen(response), 0);

        sprintf(response , "</body></html>\r\n");
        send(client_s, response , strlen(response), 0);

        close(client_s);
    }
    //close
    close(server_s);
    printf("\n\nServer shutdown\n");
    return 0;
}

 

Conclusions

I've been adding a conclusions section in my posts as of lately. Not sure what to conclude yet...

I do distrust projects like Mozilla Firefox that have fancy webpages but then deliver their code requiring a combination of software management tools or languages just for getting that code and setting up the developing software.

In this case those are Homebrew, Python and Mercurial...and their own "bootstrapping script".

And then there goes that script and tells you to install Mercurial via pip3 install Mercurial, while the webpage docs of Mozilla mention to use Homebrew for installing that VCS...

I followed the script instructions as that was running right there in my terminal. 

Well, it turns out, it's wrong. Mercurial is expected in a cellar under the Homebrew installation on your Mac.

I'm not going to bother more with Firefox. I was curious to see how complicated it would be to tweak the networking code. But it's far too much trouble: I don't feel comfortable not knowing exactly where some stuff is being installed, how easy it is to get rid of or to what extend it might mess up some system-wide configurations. Ok, you can find that out eventually. But it's not that I constantly pip install things. So it bothers me. Now I have to clean my laptop from all that was installed...

The attitude towards the bug itself as shown in those bugzilla reports and this way of delivering a development framework, that doesn't actually work, is a bit of paradigmatic of these times of fancy high-level programming in Python, Ruby, Perl....

Web Servers and the IoT

The World of embedded  Webservers

This is quick write-up of some of the things that are keeping me busy lately.

In writing some hands-on notes on networking, using as an example a server providing HTML content, I realized there might a potential market for something I called in my head Aweb. 

 Why that name? I was looking for a wordplay on webapp: a Aweb is an app that provides a webpage! ;-)

This would be but a micro/nano-webserver serving static HTML content that eventually could be even hard-coded.

Such Aweb would be extremely-narrowly dedicated. By that I mean, think of a normal webserver, say Apache, or Nginx, that would be serving only one file, or only from one folder.

It would also have a minimal storage and memory footprint...That's it! I realized then where such a thing could be useful...

Of course, embedded systems of the Internet of Things (IoT)!

Before I dig myself further into writing such a minimal webserver, I figured I look around what's already out there.

It turns out, there is a niche business for it!

But these micro/nano-webservers have a bunch of non-trivial, unique challenges. The only Wikipedia page I found is this one in french, but it gives a nice overview.

• Lost of flexibility:
• They are usually written in Assembler or C. Not surprising, and that's fine, I'm writing these servers in C  myself.
• The reuse of code is practically inexistent. This goes with the extremely targeted nature of each of these embedded servers.
  
• Portability is completely forgotten. Ditto.
  
• The skill requirements in  programming expertise are very high. I hope my upcoming hands-on on networking helps you at least with the part on networking concepts involved. :-)
• Cost sensitive:
• "The purpose of this work is to create cost-sensitive, reliable technology that enablesWeb access to distributed measurement/control systems". See here.
• Performance:
• For IoT to be successful such that people adopt it both devices as well as software must be fast enough do deliver the usability required. These applies to such a micro webserver: it has be capable of handling many simultaneous connections. Due to the resource limitations of the devices such a server is running one, this is not a trivial challenge. Power consumption is one such challenges.

Some Solutions

 

So, what is then an industry, typical such IoT hardware?

An example could be an 8-bit processor with 40KiB memory -incidentally, that reminds me the Zilog81 in the form of a Spectrum48K we had at home in my childhood: 16KiB of ROM, 48KiB of RAM!

Surprisingly enough, at least to me, efforts have been devoted to come up with an embedded, underlying, specially-tailored Java virtual environment. Such JVM would provide the missing portability. 

 

Some embedded systems use Jini, originally a Sun Microsystem's project was transferred to Apache under the name of River. 

Another Java framework used in mobile systems is J2ME, the Java Micro Edition. targeting systems with less than 100KiB of memory.

So what are some solutions out there?

 

If one can have access to similar resources a typical OS provides, such a webserver can be designed not too different from what we may do using C.

 

However, if the limitations of the physical device are too tight, that may not work. In these cases such an Aweb and the OS functionalities must be blend into the same, unique software.

• With an OS
• Contiki: From the Wikipedia page: "Extant uses for Contiki include systems for street lighting, sound monitoring for smart cities, radiation monitoring, and alarms.[...] It provides multitasking and a built-in Internet Protocol Suite (TCP/IP stack), yet needs only about 10KiB of RAM and 30 of ROM. A full system, including a graphical user interface, needs about 30 kilobytes of RAM!!". Amazing!!
• TinyOS: "operating system and platform for low-power wireless devices, such as those used in wireless sensor networks (WSNs), smartdust, ubiquitous computing, personal area networks, building automation, and smart meters. It is written in nesC, a subset of C optimized for the memory limits of sensor networks. TinyOS has been used in space, being implemented in ESTCube-1". Cool!
• Without an OS
• That Wikipedia page only mentions SMEWS (Smart Mobile Embedded Web Server). It doesn't offer more details -except that the main developer was a co-founder of Contiki.
  

The TCP/IP Stack

Here is another interesting piece. 

 

The integration of the TCP/IP stack in one such small devices isn't evident. This network and transport layer was designed in the 80's free of the limitations these new devices come with. As such a common Linux system uses of the order of several 100KiB!!

It seems the solutions go then through stripping of the usual TCP/IP stack from parts not strictly necessary in the context of IoT, or at leas that one can do without.

A guy named Tao Lin did that by removing the time-out timer of TCP. As you know, when communicating through TCP, packets for which no acknowledgment is received -within a determined time- are resent. That's at the heart of the reliability achieved by the TCP protocol. Lin hard-coded a time and considered that a packet not acknowledged within 6 secs is never going to be so and can thus be forgotten. This removes the need to resend it. 

That's interesting. Somehow, it brings us closer to an UDP protocol. In some sense, we choose the way and content of what we want to communicate. The underlying assumption is that packets are rarely lost -and if they are, we suck it up and live with it. 

 

Notice that that's what's done with video and audio: if the image freezes for sec...you really don't care to see later what the second you missed before!

This means, these kind of shortcuts drive the solutions for IoT, and hence, the whole technology towards a real-time experience, and it makes sense I guess. It's also part of our time where everything is geared towards immediacy.

 

Embbeded TCP/IP implementations are :

• µC/TCP-IP
  
• openTCP 
• lwIP : Aside from the TCP/IP stack, lwIP has several other important parts, such as a network interface, an operating system emulation layer, buffers and a memory management section. The operating system emulation layer and the network interface allow the network stack to be transplanted into an operating system, as it provides a common interface between lwIP code and the operating system kernel.

These require of the order or several 10KiB and some kind of underlying OS. (Did you notice? Somebody else also thought of the "micro" prefix. ;-). However, there are other solutions that bypass that latter requirement and access the hardware directly.

The limitation on resources means that these solutions deal only with one segment at a time and

do not manage

• network congestion management
• sorting of incoming packets
• IP fragmentation
• urgent TCP data packets.

Specific examples

An example is WebCard,  a web server for JavaCard:

• Only considers packets to port 80
• Ignores any TCP option
• Only uses HTTP1.0 not ver. 1.1
• Only implements the GET method.
• It doesn't handle IP fragmentation: it is assumed the GET request fits and arrives in one single packet!

 

I hadn't thought on IP fragmentation yet. But I was conceiving Awebs as offering only the GET mechanism indeed. 

 

Another example is MiniWeb, which is a proof of concept. It uses 24-28Bytes of RAM for the executable code and C source is a mere 400 lines long. The author doesn't mean it as more than what it is though: "It is important to note that this approach of course is totally and utterly wrong; a system using this approach should not be connected to an actual network, let alone the global Internet!".

 

It is of course limited compare to usual web servers: it handles only one connection at a time. Obviously, it doesn't offer dynamic content either.

 

Additional  technological  shortcuts are, e.g.,  client-side execution,  as is the idea of AJAX, or co-design of IoT system and software.

Another project is Ethernut: This is an Open Source Hardware and Software Project for building tiny Embedded Ethernet Devices. Firmware, hardware and (programming) tools are all developed within the same project.

This wikipedia page is a trove on low-level IoT technology. It ends with a table of examples of different embedded webservers. Notice that in french, a byte is called an octet. Thus they say kilo-octet (Ko) instead of kilobyte (KiB).

Conclusions

The low-level world of IoT is fascinating and fun. At least I'm realizing that, now that I'm writing those hands-on examples of networking code.

Maybe I have to chance to have a closer look into one of those projects.

Among those apparently more developed and/or not abandoned seem to be

• lwIP
• Ethernut
• Contiki
  

at least they are the ones that got my attention. Ok, two more did as well, but I digressed already enough on this topic for me to devote it more time. Anyway, I'm curious about microTCP and openTCP mentioned above.