Change of song

As you already know, this year I'm not devoting to much time to my projects nor the championship due to the work and the classes, and I'm advancing on ShareIt! more slowly that what I wanted.

None the less, in the last days after the exams I have been pushin WebP2P.io working on a rewrite from scratch and now has get a really clean, nice code and stable performance, and it's starting to became something really serious to be usable. My intention was to release a 1.0 version this days, but I got the conclusion that it's better to wait and get a more mature protocol, and implement from start a secure, anonimous and reliable one instead of change it in the future.

That's the reason I've decided to follow the advice of my teacher and employer Prof. Luis López and change my strategy: instead of focusing on code development, focus on the development of the ideas and write papers and specs of them. This way, I can be able to make development more dynamic and request help from others more easily (no long debuging nights stage ;-) ), instead of need to wait to have it finished (if so!) just to see that it was a waste of time, and dedicate to development when I have more time (probably summer, or maybe also the next course if I can be able to integrate my ideas in the products of my company :-) ) with the advantages of having the concepts more polished and go straight to code them.

So, at this moment I doesn't have written the specs of the new version of the protocol, but I hope to have them in some days and I'll put a post to talk about their main points. Stay tunned! ;-)

Still alive

It was almost two months from my last post, and as you can see on my GitHub account, I've didn't do any commit to ShareIt! on this time. Reason? Job and classes (and in fact I've drop out of the gym and the KinBall team during this time to focus on them...). In fact, I wanted to release ShareIt! 2.0 on the aniversary of the release of ShareIt! 1.0 "Armageddon", but it was fairly impossible, as the holidays I've taken on Las Palmas to don't get me insane shows.

Now that I have got for the first time in some years full Christmas holidays, I'll try to retake the P2P project and move it forward, although I have some pending practices for it. This is not the best year for my personal projects, but at least I hope I can finish the degree the next year, so the effort will pay off... :-)

And if not, we can always take a piece of cake... :-P

Here we go... again

Just as something that's starting to became a tradition, new course... and new inscription :-)

Yes, one more year, I'm participating (again) on the Free Software Universitary Championship, that so many good moments gave me the last year. That's the reason why I will participate again with ShareIt!, only that this time with a fairly more ambitious project (if something like this is possible!): make it a production-ready P2P framework focusing on easy to use and freedom of speech instead of just a P2P filesharing application. This has the problem that will need an almost from scratch rewrite (that I'm currently doing) isolating each area and making a lot of efforts of making them solid and future-proof, but as the existence of another WebRTC-based project on the championship shows, this kind of technologies will be the basis of the future new Internet applications on a mid-term, and it compensate the effort to give them a try. Ey, I got a job thanks to them, so they shouldn't be so bad anyway! :-D

I only hope that the job and the studies (hopefully this is my almost-last-year of college :-) ) don't kill me too much time to dedicate to the project (it's almost one month that I don't write any line of code... :-/ ).

A leap frog

In the last weeks I didn't get too much progress with ShareIt!, between exams and my new job, but specially due to the fact that with the problems I had with the removal of the anonimous XMPP server I was using and that the browsers starting with Chrome v26 generated a big SDP strings, I got fixed on using Chrome v25 if I wanted to keep experimenting :-(

But, in my new job I'm working with WebRTC all the time :-) And doing some experiments not only found that the big SDP strings was a bug of Chrome but also thanks to it another bug on Firefox has been found, but not only that, I was able to find a solution to create smaller SDP strings... and got ShareIt! working directly with Chrome v30 and their native reliable DataChannels :-D Only thing is that seems reliable support is not finished so I'm only able to get fileslist but not files content, so while it got fixed I decided to test it on Firefox and after some work... now it working there.

Yes, ShareIt! working on Chrome and Firefox!!! :-D

Well, sort of. As I said, it's mainly experimental only allowing to get files list, but they are working using native DataChannels, that's a big advance. Now I'll try to be able to use DeviceStorage API on Firefox until interoperability appear since the Filesystem API is not available nor support to upload folders, and also the input tag seems it's a dead end also in Chrome, and maybe I'll start with the Node.js based implementation ;-)

One year ago...

One year ago I didn't know what WebRTC was. In fact, if someone would talked to me about a new in-development technology to directly connect web browsers between them and that I would became some months later a international reference doing keynotes and receiving several job offers thanks to it, I would give him a pokerface :-P

But just one year ago I had a dream. I dreamed about me asking myself why browsers couldn't be able to transfer files between them in real time and how WebSockets could help on this (real time communications between a browser and a server are the main purpose of WebSockets). After this and concluding it was a good idea, I asked myself "how the browsers would meet themselves? They would need an ID"... and at this moment I got something that would be comparable to a revelation or an epiphany: just came to my head an splash of the TeamViewer call ID window...

Just have an ID for each WebSocket connected to the server and use it as an address book! EUREKA!

After that, I was not able to get back to sleep :-P

So, I had the sensation of found something great, just great enough that if I got fired from my work (something that in fact it happened just less than a week after) it only would means that it was my destiny and I should as hard as possible to develop it. So, I asked for advice to a friend of mine about create something like PasteBin but for files (something that recently has developed Pipe). He was not confident about the commercial viability of the project (it would require both ends to be connected at the same time and also there's already a lot of ways to transfer files that doesn't have this requeriment, for instance DropBox), so I decided to take another way:

- Ok, so what if I convert it on a P2P network and register on the Universitary Free Software Championship?
- If you do it, you will crush the others and win them by several heads of advantage ;-)

So I did it... and several months later I won it :-)

I'm the one in the middle with the beard and the trophy in the hand :-)

To be honest, by that time I was not really sure about how I would do it (I've never been a networks guy) nor if someone had my idea before, so I started to look for some example code to start about. That's the way I ended finding DirtyShare, a proof-of-concept whoose concept was almost similar to the mine, but that was talking about something a new technology called WebRTC and also about the WebP2P maillist. That was interesting, so looking for it, I found that it was really a better approach that the original one using WebSockets, that it was starting to being available... and that the part where I was interested, the DataChannels, was not ready yet, so that was the reason I developed DataChannel-polyfill, people started to take interest on my work and I started to forge myself a name :-D

So, this has been a really long and intense year for ShareIt! and for myself, learning a lot and coding a lot and getting more success that what I would imagine on first place :-) But this doesn't means the end of this journey, since in fact I'm still working on it and I will show ShareIt! the next course as my degree thesis thanks to the collaboration of professor Luis López. That's the way seems that the version 2.0 is not only around the corner but also will be really awesome, with a high focus on portability and modularity, code reliability (WebP2P.io, the underlying handshake framework, it's a really nice piece of cake :-D ) and easyling to integrate it on other products, expanding it to new horizonts like using it for P2P audio streaming (yeah! :-D ). Thank you to everybody that has offered to me their support and please don't move from your seats or you would loose the fast moving news that are comming around ;-)

And the winner is...

Two days late, but my mobile internet got down this weekend.

Just to say that ShareIt! has achieved the second position on the general category (first one was for Lynckia, that was using WebRTC too, by the way) and has won the Most Innovative Project category of the VII spanish national Free Software Universitary Championship 2013.

Yeah!!! :-D

Going mobile

One of the reasons why I though it would be cool to develop a P2P application inside the browser was that it could run everywhere there was a browser, without needing to install nor configure anything, just open a web page et voila! It was working. That's one of the reasons why some time ago I started to add some integration and responsive design to allow ShareIt! to work on mobile phones browsers and limited resources ones, not only natively but also as a "remote control" for a Node.js-based client, and it was fairly simple to modify the architecture and split the different components to make it possible, and also based on this architecture ShareIt! core could fully work inside a SharedWorker and being independent from the user interface (unluckily PeerConnections are not able to work inside Workers at this moment, but that's another story...). Problem was, I didn't have a mobile browsers modern enough to test it (also using WebSockets and my DataChannels-polyfill... mobile browser s*cks).

But now I have a Galaxy Note II, and can be able to run Chrome4Android for it :-) Since lately I was able to make ShareIt! work using native DataChannels, it was the perfect moment to give the mobile interface a new opportunity, and after some tune-ups on it, now I can say that ShareIt! is the first P2P application that can run on a mobile phone (and not just a proof-of-concept like the bittorrent port running on the Nokia S60 Python port... ;-) ). It need some improvements on usability and functionality (at this moment Chrome4Android doesn't support to share folders), but definitely it's almost as intuitively to use as the desktop version, and open new horizonts about what usages are possible with the WebRTC technology :-)

And if you are thinking about it... yes, DataChannels + reliable library are stable enough to be used on production, although is really slow (at least for me). I'm thinking about a solution for this like reducing packages size and wipe completely the need for reliable comunications, but I didn't take a final decision about this point...

Not so dead...

Remember my last post where I told would kill DataChannel-polyfill and work since now on native DataChannels because I got too much behind with them? Well...

...not so real :-)

Since the last post I have been thinking almost obsesively with the problem of needing a deprecated browser version just because of using my DataChannel-polyfill using WebSockets, and not working in latest versions with native support for DataChannels (that, by the way, sucks). So taking in account that the DataChannel-polyfill test code works with native ones I conclude that it would be just a problem on ShareIt! side not being fully compliant, so I went straight to it, removed all references to my polyfill and started again over with the handshake and PeerConnections layer from scratch, fixing one bug at a time.

And finally, after just some hours of work (instead of several days as I thought), it worked :-)

I must to admit that it's not the best implementation ever (I still ask my self why the h*ll is required to exchange some 'candidate' messages between the peers) and also it only support to transfer files and data up to 1Kb until they implement native reliable support (or I use reliable library, whatever comes first... :-) ), but the fact is that this almost mystical sensation after seen this thing working and with so few work (just some little tune-ups on the code thanks to the fact I develop DataChannel-polyfill trying to mimic the original specification API) it's really awesome :-D

So, not only I have experimental native DataChannels support on ShareIt! (F*ck yeah!!!!! :-D) but also being so little the diferences I've decided not only don't kill DataChannel-polyfill, but also try to improve it now that I know how native DataChannels work (and being so easy to use :-) ) and think about how to make a backward compatible API so older browsers could talk with newer ones :-)

But don't get me wrong little friends, the most important part of this post, if it was not clearly noticed yet, is the fact that now we don't need servers at all, being fully decentralized and serverless :-D Combine this with the Holy Grail of Internet that almost can bring us the HandshakeManager and the handshake servers, and wonder what an awesome future is waiting us... :-D

ShareIt! 1.1.1: Cloverfield

During the last weeks I was not able to publish too much here, specially because I'll start my exams the next week. But, since I got to be able to publish ShareIt! as my practice for my Networks-II class, I've been able to dedicate some time to it (maybe too much compared with the other asignatures... :-/ ) and develop some improvements directed by my tearcher Luis López, just in time not only for the lecture of the practice (tomorrow Monday on the morning :-P ) but also for the final phase of the championship, so here I'm showing to you the third public release of ShareIt! 1.1.1 "Cloverfield" (and no, although is the third release, the version is not written in unary ;-P)

It would be a fairly minor release because it only have two issues, but the fact is that it required some work on another ones and has added some great features and improvements on the architecture:

Manage duplicates: one of the problems that had ShareIt! previously is that since it was using the files hashes as unique identifiers of the files, if one of them was included twice only one of the copies whas showed (and served) also if it has different names. The solution was to identify them independently so both entries would coexist, and the ideal solution was to use the file's path. But what happens with the hashes? How I would be able to look for them if I'm not using them anymore as keys? I have been been putting off this thing a lot of time up to the point I didn't remember why I didn't do it before... and when I remembered it, I desired to forget it again.

IndexedDB: the reason why I have been been putting off so much the management of duplicates is because the optimal solution was to use IndexedDB's indexes, in fact a keyPath (multiple fields) for the primary key, and a non-unique index for the hashes (so they would be duplicated). What's the problem? Chrome hasn't still fixed the support of Blob and File objects inside IndexedDB (after almost one year and a half...), so I was alone using my polyfill. Two options: accept it, and told the reason to my teacher, or get mad myself, go for it and implement index support on my polyfill. At the worst, I would reuse later on my upcoming implementation of IndexedDB for Node.js for the headless version of ShareIt!... So I went directly for it, and after 8 hours listening to Daft Punk and the discography of Ke$ha and Demi Lovato (no comments), I got support for indexes on my own (in-browser, memory-only) pure Javascript IndexedDB implementation. #feelingLikeABoss... B-)

Sharedpoints: after that achievement another interesting point would have been detecting when a file has been added to a sharedpoint so it could be shared automatically without adding it by hand. Until this moment it was only able to detect when a file was deleted so it would be removed since I was using an input tag and its FileList object, that's mainly a photograph of the files that where available when the user selected it (although you can detect when it has beem removed checking an error when accessing to a file), and also it was done in a not maintable way, being all the management of the sharedpoints inside the hasher. Taking that in account, I decided to develop a new sharedpoint using DirectoryEntry from scratch, so I could be able to take in account the diferences and make it more portable. Now is fairly easier to add new types of sharedpoints like the ones I have been thinking about for Dropbox and Mega, and also I have been thinking about consider that internally the other peers would work as sharedpoints so I could simplify the way to access to them. Problem is, that I didn't notice until I finished it that Chrome has another bug not populating entries on the input tag and also I didn't be able to make drag&drop to work (this probably my fault...) so I've let this to be finished later.

And lastly by not least, the biggest feature added to this release and the one that made me think about tagging it as a mayor one, is the fact that now we have searches!!! :-D It's very primigenic and required some hacks (the fileslist is being shared between the peers after connection, having all of them the info of what are sharing others), but thanks to Lun-r.js now we can search by keywords, giving us a list of the files and hashes that correspond the them ready to be downloaded. It need some improvement in the future (like for example sending request to the others) but definitely it works :-)

But the reason why I didn't released it as a mayor release besides the fact that it has few (but important!!!) improvements, is that it will be the last of the 1.x series. This week I've discovered that ShareIt! doesn't work on Chrome browsers equal or newer than the current stable v26 because some API changes (the ones that almost crash the 1.1 release some weeks ago), so being only able to work on a legacy browser (Chrome v25) with my DataChannel-polyfill and without time in the next weeks to fix it because of the exams, I've decided to go directly to the 2.0 version after that and start using native DataChannels (I hope that by then reliable support will be implemented...), try to remove all the polyfills and develop all the backward incompatible features of the new protocol, like multiple and independent channels per PeerConnection on WebP2P or change the protocol to be RPC-based to increase reliability.

Good luck DataChannel-polyfill, and thank for the fish! :-D

WebP2P.io

I know that at this moment the WebP2P library is very tied to ShareIt!, specially because the universitary championship requires to have only one repository, but since I'll split it in the near future and in prevention of stealing it, I've decided to bought the (rather expensive...) domain WebP2P.io to host there the library project, focused in allow a server-less and distributed signaling channel for WebRTC applications. At this moment it only host a default webpage, but in the next weeks after I split the code I'll fill it with more info :-)

ShareIt! 1.1: Braveheart

Today is an epic day for ShareIt!. Lately I have been able to use the project for my networks class, and this "extra time" about working on in during my study time and also with the guidance of my teacher have lead me to make some great improvements on it, and now I'm proud to show you ShareIt! 1.1: Braveheart.

This new release not only has a lot of bugfixes, but also has a really better code structure that would allow to be easier to accept external collaborations and start adapting it to became a general purpose P2P framework, but also some great improvements on security, being the most important one the fact that now it uses by default anonimous XMPP servers to manage handshakes while being still compatible with the old system using PubNub, connect automatically to other peers on load to start creating a network mesh and speed up the connections, and not only that, now it's integrated a mobile version of the application, so if you access from your mobile phone or tablet you will be able also to use it and transfer files wherever you are. Also, all this things are ready not only for my tearcher's revision that I have in some hours that also for the keynote I'll have this afternoon at #JoinBiz10 :-D

So here you are, happy testing and don't forget that the user feedback, issues and pull requests are always welcome... :-D

ShareIt! working on anonimous XMPP (at last!!! :-D )

It was fairly more difficult to achieve that what I though at a first, but finally now it has been possible: ShareIt! is using an anonimous XMPP server to do the initial handshake instead of PubNub, so now it's not using a centralized server (it keeps the DataChannel-polyfill backend, but it's a matter of some more weeks until native support gets more stabilized) and it's using a distributed, federated network. I needed to change several things on the protocol and the implementation so now the inter-peers routing doesn't work, but I hope to fix it easily after changing WebP2P to be able to use several DataChannels at a time. This is really a HUGE step on the project viability, by the way, and just in time for the next milestone... :-D

ShareIt! on #JoinBiz10

 I have been selected to do the next monday a keynote (really, a 'lighting talk') on the #JoinBiz10, an entrepreneurship event where interesting IT projects can meet managers and MBAs and viceversa. Leets see if I can get some patrocinators for ShareIt!... :-D

And in other news, my networks teacher has accepted to use ShareIt! as final practice and has give me some guidance, and also I have just got to use XMPP instead of PubNub as handshake server. Please don't change the dial... :-P

P2P networks everywhere

A lot of people has said already that WebRTC and Peer-2-Peer comunications in the browser would be the next Internet revolution ("Internet 3.0", they call it), but I wouldn't suspect that this would start so soon: the next 7th April will be released PeerKit, the first WebRTC P2P enabled application (besides my P2P filesharing application :-P ), this case oriented as a distributed, zero cost Content Delivery Network. I don't know how effective or speed it would be (search a peer with the data, delegate the request...) and there's the problem of using the client connections to distribute the content (and with mobile phones connections this would be a serious problem), but don't doubt that if it works the server costs will drop abruptly world-wide... :-D

Slipping in the snowball

As you would know, I conceived originally ShareIt! as a real-time files-oriented PasteBin-like service using a WebSockets-based relay server (creating buzzwords the wild way :-P ). This was previously to known about WebRTC and DataChannels, that demostrated to be a way better aproach.

Obviously I wasn't the only want that saw this, and lately they are appearing a lot of other projects with a similar focus, like PeerJS, ShareFest (that recently got alpha status - congratulations! :-) ), OpenPeer or Drop&Load, and other related ones like DirtyShare, QuickShare or WhatAreYouDownloading. You could think about this that's a problem (too much related projects developed indepently and with their own incompatible protocols), and more taking in account that the WebRTC doesn't stop to being modified. What a mess! How something productive can be achieved with this perspective?

Obviously I wasn't the only want that saw this ;-) and that's the reason why in the latest weeks have been discussed to create a common P2P protocol that could make all this implementations compatible... and seems will be heavily influenced by ShareIt! 2.0 and WebP2P ideas :-D They agree on the point of split the handshake and the connection framework from the application specific protocols, so it could be easier to integrate with current protocols like Gnutella, and also it could be able to improve stability and versalite on the handshake servers (everybody is using custom ones, while it seems I'm just the only one that want to use generic third party channels... :-P).

Unluckily, I'm really busy lately with the classes up to the point I don't have time also for my work (I'll try to get up to date this spring week), so I believe I will not be able to do almost any progress on ShareIt! source code until summer, just reading others work on email. The good part is, by summer usable native DataChannels should be a reality... :-D

T3chFest Keynote

Finally, after two weeks while they were editing it, I've just received the video of the T3chFest presentation :-D


As you can see, the keynote was really well, without failures and a lot of question at the end (in fact I was almost the only one that got questions :-P ) although the limited time :-D Unluckily, I only got a third position. The reason? "It's not enought commercial, but you should think about develop it as a research project".

Taking in account that I was hearthly confident that I would get to start a company from ShareIt! and how is the situation of research on Spain, at a first point I toke the comment really bad, but on a second chance I though, isn't this was I always wanted? :-) To be honest, I've always wanted to work on R&D on my own projects, only that I need to pay the bills. That's the reason why I think that finally I'll do my career thesis with ShareIt! and work on it as an research project after that on my spare time (although it would be better to find a filantropist that believe in it! :-P ). In the last days I have been thinking about some interesting concepts for the next version of ShareIt! and WebP2P becoming them in a more general and ambitious project, and they'll need some attention and love to became true... :-)

if you can not beat them, get yourself a window and jumps

During the last six months I have been waiting for just-this-week. Why? Easy: last wednesday was the expiration day of the best domain for the project: share.it. The fact is that it was abandoned (although being a good one...) so I waited for it, but at the last moment it was renewed for one more year without any reason, and in fact currently is empty and also without a parking page. I tried to contact with it's current propietary without success, so the best option was to forget about it.

...until yesterday. I was reading the newspaper during the lunch time when I saw a full page ad from a new domain provider offering .es domains for 5€, and then got an idea: I checked out for share.it, shareit.com, shareit.net... without success, but NOT for shareit.es!!! So I quickly runned to the free-access room just to find that for inaugural promotion, the domains will be for a week at just 1€... and that the .es was free :-)

So here we are, I'm proud to show you the new ShareIt! home page, ShareIt.es:

It needs some clean-up and to fill it a little, but it a good start :-)

Also, if someone can reach me the infamous .it domain, I will thank him indefinitelly :-P

Technical issues

Trying to add support for native DataChannels on Chromium v26 I have had some problems in the last days and ShareIt! was not working correctly, so after going back and fixing it I decided to do a Draconian decission: no more references to external libraries, and use instead a local copy. I know this would break one of the points of the Universitary Free Software Championship (re-use existent code), but the fact is that the external libs are all in an unique place (the /lib folder) so it can be easily upgraded, and also this has several advantages, like being sure to have a working version all the time since it's being used an expecific version of the libraries, being easier to deploy on any other place, and since all the code is auto-contained now I don't require to add some special permissions on the FirefoxOS manifest file, so users don't need to worry about them. This also has lead me to isolate the usage of the EventTarget.js library only to WebP2P, so now the UI uses exclusively jQuery events, making it more straigforward to develop and to use less memory usage. Now it only keeps to finish a script I'm working on to update all the external libraries at a time, so the fact of being the libraries inside the source code tree or as external libs would be indiferent.

Finalist on T3chFest!

With a punctuation of 19.2 over 20 and currently in the second position on the classification (the first one is for an universitary social network, that being the truth, I would love to see it being used as my university virtual campus... :-D ), I've been finalist on the T3chFest and the next friday I will present ShareIt! in front of a crowd of 70 persons and also compete for the final points for the first position :-D It will be recorded and also streamed, so I will upload the video when I get it :-)

A little bit more work, and the next step the final of the national Universitary Free Software Championship! :-D

When ShareIt! met Gnutella...

On the recent post on HackerNews has appeared a curious comment:

Hmm, it sounds like you're reimplementing Gnutella for the browser, which is not necessarily a bad thing!

I've never been a P2P guy since I was connecting to Internet using neigtbours wifi (and it was already dificult to do just plain e-mail and web surfing...), so the other times I read about how the diferent P2P protocols worked I only got the more superficial idea. That's the reason why this comment got my interest and take a look on wikipedia.

But this time it was diferent: I'm developing my own P2P protocol, so now I know what all that words means, and maybe just a trick of my unsconscious or maybe just serendipity, but I got a good surprise... both protocols and priorities ARE ALMOST THE SAME!!!! :-D This have two direct implications:

1. ShareIt! and the WebP2P protocol are not a total innovation "of my own", so this would be a little bit annoying (the "not invented here" complex... hey, what's the matter? I was raised using Macintosh... :-P )
2. on the other hand, having both projects the same concepts in mind I could learn from their errors, and also adapting the protocol to use theirs and creating proxies in the same way of the SIP-over-WebSockets ones, I would be able to access to the currently in active most used P2P network in the world by free :-D

I don't know, but by the moment, I'll try to send an email to the Gnutella guys to see what's happens... :-P