I implemented the decompression algorithm in Lua 3.2, but it takes too long to complete the process. I do not understand why this happens. But data compression is extremely fast.

[Compress / Decompress] Python

[Decompress only] Lua 3.2

This implementation/upgrade of LZW is genius!! Specially for Chrome since the localStorage stores every single byte as a word (2 bytes) occupying twice as much space hence instead of having 5MB of storage it gets reduced to 2.5MB.

C'est génial!

hi

Great job !!!

In my application I would like get from server compressed template data. Do you think about implementing this algorithm in java?

FYI, I'm using LZString within a chrome-app (aka packaged app) and found that while using LZString.compress() to compress ~12MB of string before saving in chrome.storage.local, Chrome started suspending my chrome app unpredictably. Switching to LZString.compressToUTF16() fixed the suspend problem.

I suspect Chrome has some background storage checker that suspends extensions that store malformed data, like LZString compressed strings.

chrome.storage.local is storage API available to Chrome apps and extensions.

You can find documentation here: http://developer.chrome.com/apps/storage.html

Unlike localStorage, it's async. It's still limited to 5MB.

Good news is that I've managed to cut down my data to ~3.7M *characters* which,

using LZString compresses down to 10% (w00t). Bad news is, when I call

chrome.storage.local.getBytesInUse, returned number is ~2.5M *bytes* which,

if correct, suggests chrome.storage.local implementation doesn't just store what it

receives but transforms it somehow such that much of the compression magic gets

diffused. It could be that localStorage does the same.

 Thanks for the effort. Just out of curiosity, why don't you rely on the webserver compression? That one is full fledge gzip which ought to compress a lot better than LZ-String...

Anyways, I'll link to your project from the documentation.

I am experimenting with lzString to store compressed data from javascript simulations

LZString seems to be working fine

But... when I bring up the debug tools, and click over to the Local Storage tab, it hangs and

the debug tools can not be dismissed with the X.

The website keeps working though.

Local storage with uncompressed data, even though it is pretty big (250K chars), does not

break the chrome Local Storage viewing tab in the dev tools.

This is probably  chromium's problem, not yours.  I have not yet tried other browsers.

Just thought you might want to be aware of it, and wondering what it might be....

 You're an absolute life saver, extremely easy to use and opens up lots of possibilities with cookies.

I'm testing a node client <--> server chat program, and was wondering if your script was working for sending base64Strings.
It is working and it is not working :S ;)

A simple test gives me this:

var stringex = "This is my compression test.";
console.log("Size of sample is: " + stringex.length);

var compressed = Base64String.compress(stringex);
console.log("Size of compressed sample is: " + compressed.length);
string = Base64String.decompress(compressed);
console.log("Base64String Sample is: " + string);

var compressed3 = LZString.compressToUTF16(stringex);
console.log("Size of compressToUTF16 sample is: " + compressed3.length);
string = LZString.decompressFromUTF16(compressed3);
console.log("compressToUTF16 Sample is: " + string);

output:

Size of sample is: 28
Size of compressed sample is: 10
Base64String Sample is: ThisismycompressiontestA
Size of compressToUTF16 sample is: 17
compressToUTF16 Sample is: null

I just released version 1.3.3. Version 1.3.2 was the result of a pull request that I obviously didn't test enough...

 That's the idea. How will you get the resulting encoded string on the client side? Ajax.responseText ?

I'm not worried about encoding problems here, but I am worried that a UTF-8 encoded string is going to be substantially bigger than the UTF-16 counterpart LZString is producing, resulting in wasted bandwidth. Your UTF16 detection is going to return true all the time : compressToUTF16 *is* generating UTF-16 characters as its name can tell.

The best way would probably be to generate ISO-(LATIN-1 for example) characters (all 256 of them being valid), or UTF-16. But for this to be "optimal" you need the content type to be set correctly. Do you have the hand on the content encoding of these requests? If yes, I suggest switching to "Content-Type: text/html; charset=utf-16" for a better bandwidth usage. 

If not, we'd need to write a compressToUTF8, using 7 bytes per character. Not quite hard.

The ideal solution would be for socket.io to be able to transfer byte arrays instead of strings. After all, we're trying to transmit binary data, not text.

 You are confused between two very different things: compression and encoding. 

Compression is the act of taking bits as an input and outputing less bits on the output. This is where the LZ part of LZString is working.

Encoding is the act of taking bits as input and outputing characters as output. This is where the String part of LZString is working. This is needed because JavaScript doesn't know (consistently at least) how to handle binary data. It can handle numbers and strings. 

So no, you cannot "make more compression" using compressToUTF8(). compress returns to you a UTF-16 string that is packed at the maximum but all 65536 values of a 16-bit int are not valid, so the string is indeed an invalid UTF-16 string. This is why I created compressToUTF16 that will only store 15 bits per character (hence, a slightly less optimal encoding) but produces a valid string. compressToBase64 gives you a string where only 6 bits are used for each characters. You might think that's complete bullshit as the string will be more than twice as big as a string produced with compressToUTF16, and yet is it the most efficient way to upload your data to your server, because everything will pass through the "url encoding" encoder, making every character outside the Base64 range three bytes at least.

Now, to exchange data, computers usually use bytes, not strings. UTF-8, ISO-8859-1 and UTF-16 are methods (encodings) to represent a string as a byte stream. A 1024 characters string may be represented as 1024 bytes in ISO-8859-1 but 2048 bytes in UTF-8. Similarly, another string may be represented as 1000 bytes in UTF-8 but 2000 bytes in UTF-16. Another, 1000 bytes in UTF-16 and 1500 bytes in UTF-8. So you have to know what you're doing in order to optimize your stream of data. Just calling "length" on your strings gives you the number of characters, not the number of bytes needed to tranfer thesse characters to your browser.

So, yes, anyone can write a compressToUTF8() method (it is actually very simple). But in UTF-8, the first bit is reserved for higher characters. So you can only store 7 bits per characters, wasting a full 12.5% of bandwidth, where with UTF-16 I only waste half of that to be UTF-compliant. I am sure you can set the content-type of your requests with your library, so why not just setting it up to be UTF-16 and be over with it? You save bandwidth and time, to get a more elegant solution.

But if you use compressToUTF16 and send it out encoded in UTF-8, your stream may very well end up being twice as big as it needs to be. Again, only testing will tell.

Thanks for helping me with this haha, its hard to understand it :)

What i'm testing right now is this:

Sending message function(){

msg = JSON.stringify(args[1]);
compres = LZString.LZString.compressToBase64(msg);

if(Buffer.byteLength(compres, 'base64')<Buffer.byteLength(msg, 'utf8')){

<span class="Apple-tab-span" style="white-space:pre"> </span>console.log('COMPRESSION');

       args = ['event',{compr:compres}];

}

}

http://nodejs.org/api/buffer.html#buffer_class_method_buffer_bytelength_string_encoding

So it checks if the compression is smaller then the utf-8 message.

 <span style="font-family: 'Trebuchet MS', 'Lucida Grande', Tahoma, Verdana, Arial, sans-serif; font-size: 16px; line-height: 24px; background-color: rgb(240, 240, 240);"> Wow... I can see quite a few mistakes here. First, you compress your content and then encode it in Base64 twice... Once with compresstoBase64 and once with Buffer.byteLength(..., 'base46'), hence, multiplying by ~1.8 its size. Looks like you need to read a bit about what is an encoding and what is compression.</span>

?? <span style="background-color: rgb(240, 240, 240); font-family: 'Trebuchet MS', 'Lucida Grande', Tahoma, Verdana, Arial, sans-serif; font-size: 16px; line-height: 24px;">Buffer.byteLength(..., 'base46')</span> gives me the compression length
<span style="font-family: 'Trebuchet MS', 'Lucida Grande', Tahoma, Verdana, Arial, sans-serif; font-size: 16px; line-height: 24px; background-color: rgb(240, 240, 240);">Buffer.byteLength(msg, 'utf8') gives me the non-compression length</span>

If the compression is smaller then I send the compressed data to the client using socket.io (websocket/flash/ajax/etc)
If the compression is not smaller then I send the non-compressed data to the client using socket.io

Socket.io don't use <span style="background-color: rgb(240, 240, 240); font-family: 'Trebuchet MS', 'Lucida Grande', Tahoma, Verdana, Arial, sans-serif; font-size: 16px; line-height: 24px;">gzip for the messages.
So that's why i'm searching for a compressed method to send the messages to the clients and back using javascript.
I works now great, with some small compression in some text messages.</span>

• Compression on the server looks rather useless here. gzip is built in every server and browser out there, performs better and should be used when going from the server to the client.
• The decompression routine is much faster than the compression one
• The lib is meant for small amounts of data, and its dictionary isn't capped. So with large chunks of data it may take a lot of memory. 200kb looks very low in this regard and I have compressed much larger chunks of data in browsers in a timely fashion.
• I have a port of lz4 in javascript coming but I really have no timeline for it. I may be able to work on it a little more in the coming weeks (holiday season) but I make no promises.

Great to hear you found out about the issue. Thanks for the update.

Base128 really is a corner case, so I don't think it should be incorporated in the lib itself. It can be incorporated as an "add-on" for extra encoding options. You can submit a patch for a file called "lzstring-encoding.js" for example. I might add some other URL encoding as well in there. That sounds like a good idea.

Have you tried this? Are you interested in taking the code back if I make the necessary modifications?

I'm looking at including it in an application where we will use the Closure compiler to minify and concatenate it with other code.

This has nothing to do with node.js and I don't know whether the last three (3) lines are relevant, yet.

I should have just compressed it first, however, our experience has been that code that passes JSLint always works when minified and code that doesn't regularly - though not always - has issues that are not always obvious.

Our standard has been to JSLint all code to increase our odds that all code paths are Ok, not just the ones we've managed to test (min'd vs orig).

I'd still encourage you to JSLint the source as it will minimize the chances there is code that will not minify gracefully.

Last question: Is this on a page on the internet where I can have a look?

compressToBase64().replace("=","$").replace("/","-")

Just encode your JPEG in base46 and send it to your server.

That said, the compress method is pretty much useless since there's very little one can do with the result. So cloning the algo for compressToUTF16 probably makes sense. I'll get into that. Thanks.

It looks as if you've ported the latest version. You should really NOT port the latest version which is full of bad stuff to make the various JS engines happy.

Can you try porting version 1.0.2? This will be much much easier to debug (and easier to write). Here it is

Good! Now what we need to figure out if it is the decompression or the compression that doesn't work.

You have two strings: one that works and one that doesn't. Here is the result of the compression in Base46 of both your strings:

LZString135.compressToBase64("hello1hello2hello3hello4hello5hello6") => "BYUwNmD2CMoZAmOUDMzIBZ0FZ0DYgA=="
LZString135.compressToBase64("hello1hello2hello3hello4hello5") => "BYUwNmD2CMoZAmOUDMzIBZ0FYgA="

Can you post here the result of the compression of both strings by your library (in base64)?

I assume both previous comments were from you. What you need to check is that the String you are feeding decompressFromBase64() on the C# side is the exact same as the string you are getting from compressToBase64() from your browser. My guess is that they are not the same, because some of the characters in base64 need to be URL encoded.

You should use compressFromEncodedURIComponent(). This is like base64 but the characters chosen are slightly different (see the last two chars below). With a couple of replaces on the server side - String.replace("-","/") and String.replace("$","=") - you should get back the original base64 string. Just feed that to decompressFromBase64() in your C# app.

The characters used for base64: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=

The characters used for uricomp: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-$

If you still have issues, please open a bug into the C# repo (I'm not the author of the C# code)

compress8=function(uncompressed){
  //return LZString._compress(uncompressed,16,function(a){return String.fromCharCode(a);});
  return LZString._compress(uncompressed,8,function(a){return String.fromCharCode(a);});
}

decompress8=function(compressed){
  if (compressed == null) return "";
  if (compressed == "") return null;
  //return LZString._decompress(compressed.length,32768,function(index){return compressed.charCodeAt(index);});
  return LZString._decompress(compressed.length,128,function(index){return compressed.charCodeAt(index);}); 
}

The answer is in the switch(c). If the decompressor receives an entry 0, 1 or 2 it has a special meaning. All other entries IDs are to be taken from the dictionary. 0 means it's followed by an 8 bit entity. 1 by a 16 bit entity. 2 means it's the end of the stream. So put whatever you want in the first three entries, the code will never read from it. Or don't put anything. But start appending to your dictionary at index 3.

Let me know if you still have issues.

// Copyright © 2016 Gary W. Hudson Esq. All rights reserved
var lzw = (function() {var z={

Decode:function(p)
{function f(){--h?k>>=1:(k=p.charCodeAt(q++)-32,h=15);return k&1}var h=1,q=0,k=0,e=[""],l=[],g=0,m=0,c="",d,a=0,n,b;
do{m&&(e[g-1]=c.charAt(0));m=a;l.push(c);d=0;for(a=g++;d!=a;)f()?d=(d+a>>1)+1:a=d+a>>1;if(d)c=l[d]+e[d],e[g]=c.charAt(0)
else{b=1;do for(n=8;n--;b*=2)d+=b*f();while(f());d&&(c=String.fromCharCode(d-1),e[g]="")}}while(d);return l.join("")},

Encode:function(p)
{function f(b){b&&(k|=e);16384==e?(q.push(String.fromCharCode(k+32)),e=1,k=0):e<<=1}function h(b,d){for(var a=0,e,c=l++;a!=c;)
e=a+c>>1,b>e?(a=e+1,f(1)):(c=e,f(0));if(!a){-1!=b&&(a=d+1);do{for(c=8;c--;a=(a-a%2)/2)f(a%2);f(a)}while(a)}}for(var q=[],k=0,
e=1,l=0,g=[],m=[],c=0,d=p.length,a,n,b=0;c<d;)a=p.charCodeAt(c++),g[b]?(n=g[b].indexOf(a),-1==n?(g[b].push(a),m[b].push(l+1),
c-=b?1:0,h(b,a),b=0):b=m[b][n]):(g[b]=[a],m[b]=[l+1],c-=b?1:0,h(b,a),b=0);b&&h(b,0);for(h(-1,0);1!=e;)f(0);return q.join("")}

};return z})();
if(typeof define==='function'&&define.amd)define(function(){return lzw})
else if(typeof module!=='undefined'&&module!=null)module.exports=lzw;