Thanks.

I suspected there was a relevant RFC or two available somewhere. Some months ago I did look into the situation on standard log file syntax and do have some notes.

> The problem is that in Windows it is not the standard, so if you want
to send log messages to a syslog server in Linux from a web server
running in Windows, along with your own messages you will have to
configure a “syslog agent”.

I am rendering unto Windows what is Windows and not trying to do anything with real, official, Gates-blessed, Windows log messages or the Microsoft tools for such messages.

Instead I’m just trying to handle just messages from my running application level code, messages my code writes.

Here is a sample message my software wrote by calling the Windows logging tool, that is, my message after I ripped out the gibberish Windows put in:

20160227 042015.787000 UTC Universal 0.0.0.0 search_inputs.aspx SSS_retrieve 1002 Unexpected condition. return_code = 1002 Exception thrown. error_code = 1003 err.number = 5 err.source = mscorlib err.description = End of Stream encountered before parsing was completed.

That message and a glance at the code, with my voluminous comments, will give a good explanation of just what the heck happened.

Maybe I’ll glance at the RFC and other standard stuff, but I know basically what I want — some fixed fields followed by some name-value pairs with blanks for delimiters, just the first 128 or so ASCII characters, and no gibberish.

For sending the message, I will just use my own code, and it will use just simple as possible TCP/IP sockets. Really, my server farm log file server won’t be using any Windows code that is specific to log files.

For UDP, sure, that would be sufficient and more efficient, but, still, I’ll just use TCP/IP anyway if only because it’s my general purpose, workhorse means.

Sure, a Linux box connected to the serer farm LAN could receive the messages just fine. Heck, in principle could use a Raspberry Pi with a TCP/IP stack and thumb drive! The log file server just receives a string via TCP/IP and writes it to a file — the string could be a recipe for chocolate cake as far as the server is concerned — so, anything with a TCP/IP stack that could write a file and run a little program could be used as the log server. But, I’m working with Windows so don’t want the overhead of also working with Linux. I picked just one of those two, Windows, and not both.

Sure, in a Web server, say, Microsoft’s IIS, the Windows code did me a nice service: There could be at one time several Web pages using the Windows tools to write to the Web site log. So, Microsoft had to serialize the writing so that each message got written without being interrupted by messages from other Web pages.

So, what will my log server do about several executing program instances, each on its own thread all wanting to write to the log at the same time? Sure: At the log server, run that code all on just one thread and set the TCP/IP input queue to a relatively high number, say, 100 messages, and use the TCP/IP FIFO mechanism to handle the serialization, contention, whatever want to call it. If that fails, e,g., a sender trying to connect gets a TCP/IP timeout, then I’ll think of a patchup! E.g., a program getting such a timeout should report it to its own console log or some such.

Thanks.

syslog, RFC 5424 compatibility is a nice design goal because you gain some benefits, as the third party analytics functionality you mention. The problem is that in Windows it is not the standard, so if you want to send log messages to a syslog server in Linux from a web server running in Windows, along with your own messages you will have to configure a “syslog agent”. Sending messages to a syslog server from your code is relatively easy because it is basically an UDP send.

Thanks.

I’ve tried to stick with the first 128 or so characters of ASCII and ignore Unicode and have mostly been successful!

It’s good that Microsoft has done the good and right things to support Unicode, and if my startup goes international or otherwise has to handle a lot more than those first 128, then I will thank Microsoft.

In places Windows is willing to write a file of its options; looking at that file with my favorite ASCII editore, KEdit, shows gibberish characters; but looking at the file with Microsoft’s Notepad gets rid of the gibberish and shows just clean text and numerical data. Then SELECT all that content, copy it to the system clipboard, pull it into my favorite ASCII editor — still gives clean content without gibberish characters. I have successfully avoided looking into just why.

Well, with your suggestion, I just went to my file system directory with lots of log files from the Windows tools for writing logs (since all my planned code does run on my development computer and I have run it many times so have lots of log files) and had Notepad display one of the files. Yup, even Notepad displayed the little rectangles indicating gibberish characters. I looked at the file in hex and saw that the rectangles were from hex 10 and other relatively small numbers, in ASCII intended for controlling old serial telecommunications equipment going way back, maybe to teletypes in the 1930s.

I suspect that these gibberish characters are being used as delimiters in some syntax — maybe somewhere there is come Bachus-Naur Form (BNF) syntax for such log files.

Whatever: I need a good log file facility, tool, function, server, whatever call it. Microsoft is not so good on documenting the syntax they use, and I don’t want to try to reverse engineer what they are doing by gussing, throwing at the wall, and stopping as soon as something appears to stick — I’ve done such things too often.

And there’s no biggie problem: I can just set up a little single threaded program that receives communications via simple TCP/IP sockets. Each such message will have just the first 128 ASCII characters, one per 8 bit byte. The BNF will be dirt simple — a sequence of tokens where each token is a string without blanks and delimited by blanks. But if I get too wound up, I may let a token be delimited by double quote marks, , [ and ], or { and } — bummer, overkill, leave for later where the simple syntax is a subset of the overkill one! But long ago I wrote code for the overkill syntax and use it for reading ASCII files of input options for programs. It works fine. So, I might reuse that old code. Nope — just use KISS, keep it simple, stupid!

That KISS solution should be good enough for the first $1 B in company value; if the KISS solution is not good enough for the first $1 T, then cross that bridge then!

Besides, the BNF is just for the lexical parsing and syntax; I can still use reserved tokens for implement desired semantics!

So, each line of the log file should have one log message — no multiline log file messages permitted! If a string sent to the log file contains LF or CR characters, then that is a misuse of the facility!

Each message starts with time and date, Greenwich Mean Time (GMT) or Coordinated Universal Time (UTC) depending on how correct want to call it. So, presto, bingo, don’t have to worry about missing hours or clocks jumping backwards due to daylight savings time! And have the time data as a token with no blanks, all tokens the same length, and so that sorting as ASCII characters also sorts in time sequential order!

Then have tokens for the executing instance of the program, the name of the program, the function being executed, and the sequential number of the message in that function (uh, my favorite text editor has a little command that sequentially numbers such code components!) — so, have the origin of the message uniquely identified. Then have more specific data, say, from a Web server, the IP address of the user, the user ID (UID) if are having the users login. Then have the rest of the message — evidence of a hacking attempt, unexpected condition in the software, software error, routine report of some possibly useful data, maybe for analyzing the UI/UX, etc.

In the code for all my Web pages, to write to the log file, I call the same function I wrote with the same list of parameters (call signature). So, just keep that function name and the uses of it and, then, just
change the function to send the data to the log
file server instead of calling Microsoft’s function. Maybe more
work to describe here
than to implement the
thing!

Sure, for scalability, should
want an easy way to
have more than one
instance of the log file
server running! There should be an easy way to do that.

And, sure, just one such server would be a
“single point of failure”
for the whole Web site
and a reliability threat. So, implement an automatic facility to switch over to a backup log file server — do that later!

So, net, “Look, Ma, no gibberish characters in the site log files!”.

Thanks.

Happy Thanksgiving, Dr. Sigma. Best wishes.

Many thanks. About to leave for some groceries and a dinner — I’ll pick up an apple, cherry, or pumpkin pie!

My family did a big deal on turkey day, and I did in my marriage. Used Dad’s turkey recipe he got from his mom who used very lean wild turkeys so used a bread stuffing with a lot of fat, onions, stock, etc. At the end of the roasting, the bottom of the roasting pan had lots of good base for gravy!

Applying Dad’s recipe to a modern, say, Butterball turkey, is real overkill but really good! And I used to tweak the gravy a little to have it more like what I learned about French sauce making — reduce the liquid to concentrate flavors, maybe with some dry white wine, use some of the usual suspects, garlic, bay leaf, parsley, strain it, make a white roux using the fat from the roasting pan, combine and make a thick gravy, thin with milk or cream, adjust salt and pepper.

Once after Turkey day, made a stock from the turkey bones and was boiling it down. Had the back door open. In wandered a hungry kitty cat. She was with us for years!

Once after turkey day, Dad and I got up about 2 AM and drove to Reelfoot Lake in NW TN for duck hunting. For lunch, we make sandwiches out of left over turkey. Wrapped the sandwiches in AL foil. In the duck blind, for heat, we had some charcoal in a left over five quart can. So, to warm the sandwiches for lunch, just set them in the foil on the charcoal. GREAT sandwiches! That some parts were burned was okay! Ah, maybe how good the sandwiches were had something to do with standing most of the morning in that duck blind!

It’s corny but still pretty good: In a casserole dish of, say, 1 quart put a piece of toast. Top that with a thin slice of good ham, maybe smoked. Cube some of the roast turkey. Top with some of the turkey gravy (can’t have too much turkey gravy!). Warm that in the microwave and maybe brown the top under a broiler. Can make a nice main dish.

Many thanks.

Happy Thanksgiving for you too Jeff!

We don’t celebrate it explicitly as you up there, I guess our December 24 dinner resembles all the spirit of your celebration. My greetings to you goes with that spirit.

Happy Thanksgiving! ,

Happy Thanksgiving!

Those gibberish characters may be that you are looking at them from a different encoding, have you checked that? Just a few days ago I was processing what I thought to be an ASCII file, but things were not working as expected. Turned out that it was UTF-8 encoded which is variable length encoding in a way that a clean ASCII file can be considered UTF-8 because ASCII is a subset of UTF-8. When I corrected that all the reported ‘errors’ disappeared magically.

Yes, delaying Version 1.0 too long is easy. But there is also something easier too many people know too much about, learned by “paying full tuitioin” — simple, ordinary, failure. Want success. Certainly want to avoid failure.

Concern 1:

You only get one chance to make a good first impression.

Or, similarly, “Measure twice. Saw once.”

Concern 2:

To go from 0 to more, to the first 1000 customers, to make progress at all, may have to get publicity, say, via a writer in the tech media. For that, the writer has to be impressed enough to write an article. So, Version 1.0 has to be good enough to impress a tech writer.

Concern 3:

Supposedly there are a lot of people nearly no one has heard of who had and still have a better mousetrap and, thus, don’t believe the common

Build a better mousetrap, and the world will beat a path to your door.

Instead, have to do well on launch excitement, buzz, both steak and sizzle.

Alpha test is important: That’s where detect and fix the worst of the bugs, rough edges, etc.

Beta test is important: That’s where get real, that is, blunt, objective, feedback from real target users, likely nearly all strangers.

If want to get equity funding, then early, rapid growth is important (likely nearly the only thing early stage funders look for), and one of the best sources is viral growth. So, if have a feature that stands to do good things for viral growth, implement that in Version 1.0 or wait? Hmm ….

Concern 4:

There is

If you fail to plan, then you plan to fail.

So, maybe there is the best news: The users are thrilled and arrive by the thousands. Then, suddenly, for some unexpected reason, there is some big publicity, and then users arrive by the millions.

So, don’t want this good news to kill the business. So, want a way to throttle this success, e.g., have users need a user ID (UID) and password (PW), apply for those, and wait in a FIFO queue. So, need to be able to implement that feature during the mad rush or have it working by, say, beta test and turn it on if needed during Version 1.0.

Also want user feedback e-mail and advertiser feedback e-mail.

Here’s one: The Web server and, really, the rest of the server farm need a log file. Okay, Microsoft has some code for that. Of course, at a live Web site that is at all successful, the log file can get some thousands, maybe millions, of lines of data a day. But the Microsoft code is for a little Microsoft viewer program where get to look at a few dozen log file lines at a time. Bummer. Yes, can write some code to parse that log file and write some code to analyze the parsed data, but Microsoft doesn’t describe the contents; there are a lot of gibberish characters; so have to throw away the gibberish characters, etc.

So, really should have an okay log file server for the whole server farm. Then, sure, likely don’t do much with the file unless and until there is some serious need, but at least do have the log file server, have the server farm production software sending log messages to it, and have the server writing the messages to a disk file, yes, with an option to tell the log file server to close that file and start a new one.

Sure, the Web server(s) need a session state server. So, if have one, say, with just TCP/IP socket communications, sending/receiving serialized versions of instances of classes, using, say, two instances of a collection class to store the session states and keep track of their delete times, then just copy that code, rip out most of it, put in a few lines to write to a file, modify the feature that lets the code receive real time commands and respond with messages, and, presto, bingo, have a server farm log server. Worth doing it before Version 1.0, likely before beta test.

Ah, sounds like when to take “that big step off” to Version 1.0 needs some judgment! Gee, that business might need some judgment! Amazing!