Still alive

It was almost two months from my last post, and as you can see on my GitHub account, I've didn't do any commit to ShareIt! on this time. Reason? Job and classes (and in fact I've drop out of the gym and the KinBall team during this time to focus on them...). In fact, I wanted to release ShareIt! 2.0 on the aniversary of the release of ShareIt! 1.0 "Armageddon", but it was fairly impossible, as the holidays I've taken on Las Palmas to don't get me insane shows.

Now that I have got for the first time in some years full Christmas holidays, I'll try to retake the P2P project and move it forward, although I have some pending practices for it. This is not the best year for my personal projects, but at least I hope I can finish the degree the next year, so the effort will pay off... :-)

And if not, we can always take a piece of cake... :-P

Here we go... again

Just as something that's starting to became a tradition, new course... and new inscription :-)

Yes, one more year, I'm participating (again) on the Free Software Universitary Championship, that so many good moments gave me the last year. That's the reason why I will participate again with ShareIt!, only that this time with a fairly more ambitious project (if something like this is possible!): make it a production-ready P2P framework focusing on easy to use and freedom of speech instead of just a P2P filesharing application. This has the problem that will need an almost from scratch rewrite (that I'm currently doing) isolating each area and making a lot of efforts of making them solid and future-proof, but as the existence of another WebRTC-based project on the championship shows, this kind of technologies will be the basis of the future new Internet applications on a mid-term, and it compensate the effort to give them a try. Ey, I got a job thanks to them, so they shouldn't be so bad anyway! :-D

I only hope that the job and the studies (hopefully this is my almost-last-year of college :-) ) don't kill me too much time to dedicate to the project (it's almost one month that I don't write any line of code... :-/ ).

Teaching the hard way

When we teach kids to ride a bike, at some point we have to take the training wheels off. Here’s an idea. When they hit eleven, give them a plaintext file with ten-thousand WPA2 keys and tell them that the real one is in there somewhere. See how quickly they discover Python or Bash then.

EPIC

A leap frog

In the last weeks I didn't get too much progress with ShareIt!, between exams and my new job, but specially due to the fact that with the problems I had with the removal of the anonimous XMPP server I was using and that the browsers starting with Chrome v26 generated a big SDP strings, I got fixed on using Chrome v25 if I wanted to keep experimenting :-(

But, in my new job I'm working with WebRTC all the time :-) And doing some experiments not only found that the big SDP strings was a bug of Chrome but also thanks to it another bug on Firefox has been found, but not only that, I was able to find a solution to create smaller SDP strings... and got ShareIt! working directly with Chrome v30 and their native reliable DataChannels :-D Only thing is that seems reliable support is not finished so I'm only able to get fileslist but not files content, so while it got fixed I decided to test it on Firefox and after some work... now it working there.

Yes, ShareIt! working on Chrome and Firefox!!! :-D

Well, sort of. As I said, it's mainly experimental only allowing to get files list, but they are working using native DataChannels, that's a big advance. Now I'll try to be able to use DeviceStorage API on Firefox until interoperability appear since the Filesystem API is not available nor support to upload folders, and also the input tag seems it's a dead end also in Chrome, and maybe I'll start with the Node.js based implementation ;-)

One year ago...

One year ago I didn't know what WebRTC was. In fact, if someone would talked to me about a new in-development technology to directly connect web browsers between them and that I would became some months later a international reference doing keynotes and receiving several job offers thanks to it, I would give him a pokerface :-P

But just one year ago I had a dream. I dreamed about me asking myself why browsers couldn't be able to transfer files between them in real time and how WebSockets could help on this (real time communications between a browser and a server are the main purpose of WebSockets). After this and concluding it was a good idea, I asked myself "how the browsers would meet themselves? They would need an ID"... and at this moment I got something that would be comparable to a revelation or an epiphany: just came to my head an splash of the TeamViewer call ID window...

Just have an ID for each WebSocket connected to the server and use it as an address book! EUREKA!

After that, I was not able to get back to sleep :-P

So, I had the sensation of found something great, just great enough that if I got fired from my work (something that in fact it happened just less than a week after) it only would means that it was my destiny and I should as hard as possible to develop it. So, I asked for advice to a friend of mine about create something like PasteBin but for files (something that recently has developed Pipe). He was not confident about the commercial viability of the project (it would require both ends to be connected at the same time and also there's already a lot of ways to transfer files that doesn't have this requeriment, for instance DropBox), so I decided to take another way:

- Ok, so what if I convert it on a P2P network and register on the Universitary Free Software Championship?
- If you do it, you will crush the others and win them by several heads of advantage ;-)

So I did it... and several months later I won it :-)

I'm the one in the middle with the beard and the trophy in the hand :-)

To be honest, by that time I was not really sure about how I would do it (I've never been a networks guy) nor if someone had my idea before, so I started to look for some example code to start about. That's the way I ended finding DirtyShare, a proof-of-concept whoose concept was almost similar to the mine, but that was talking about something a new technology called WebRTC and also about the WebP2P maillist. That was interesting, so looking for it, I found that it was really a better approach that the original one using WebSockets, that it was starting to being available... and that the part where I was interested, the DataChannels, was not ready yet, so that was the reason I developed DataChannel-polyfill, people started to take interest on my work and I started to forge myself a name :-D

So, this has been a really long and intense year for ShareIt! and for myself, learning a lot and coding a lot and getting more success that what I would imagine on first place :-) But this doesn't means the end of this journey, since in fact I'm still working on it and I will show ShareIt! the next course as my degree thesis thanks to the collaboration of professor Luis López. That's the way seems that the version 2.0 is not only around the corner but also will be really awesome, with a high focus on portability and modularity, code reliability (WebP2P.io, the underlying handshake framework, it's a really nice piece of cake :-D ) and easyling to integrate it on other products, expanding it to new horizonts like using it for P2P audio streaming (yeah! :-D ). Thank you to everybody that has offered to me their support and please don't move from your seats or you would loose the fast moving news that are comming around ;-)

Achievement unlocked

Banned from W3C Restricted Media mail list for being critic with EME & DRM on HTML5. I love to be on the first line of fire... >:-D

And the winner is...

Two days late, but my mobile internet got down this weekend.

Just to say that ShareIt! has achieved the second position on the general category (first one was for Lynckia, that was using WebRTC too, by the way) and has won the Most Innovative Project category of the VII spanish national Free Software Universitary Championship 2013.

Yeah!!! :-D

Going mobile

One of the reasons why I though it would be cool to develop a P2P application inside the browser was that it could run everywhere there was a browser, without needing to install nor configure anything, just open a web page et voila! It was working. That's one of the reasons why some time ago I started to add some integration and responsive design to allow ShareIt! to work on mobile phones browsers and limited resources ones, not only natively but also as a "remote control" for a Node.js-based client, and it was fairly simple to modify the architecture and split the different components to make it possible, and also based on this architecture ShareIt! core could fully work inside a SharedWorker and being independent from the user interface (unluckily PeerConnections are not able to work inside Workers at this moment, but that's another story...). Problem was, I didn't have a mobile browsers modern enough to test it (also using WebSockets and my DataChannels-polyfill... mobile browser s*cks).

But now I have a Galaxy Note II, and can be able to run Chrome4Android for it :-) Since lately I was able to make ShareIt! work using native DataChannels, it was the perfect moment to give the mobile interface a new opportunity, and after some tune-ups on it, now I can say that ShareIt! is the first P2P application that can run on a mobile phone (and not just a proof-of-concept like the bittorrent port running on the Nokia S60 Python port... ;-) ). It need some improvements on usability and functionality (at this moment Chrome4Android doesn't support to share folders), but definitely it's almost as intuitively to use as the desktop version, and open new horizonts about what usages are possible with the WebRTC technology :-)

And if you are thinking about it... yes, DataChannels + reliable library are stable enough to be used on production, although is really slow (at least for me). I'm thinking about a solution for this like reducing packages size and wipe completely the need for reliable comunications, but I didn't take a final decision about this point...

Not so dead...

Remember my last post where I told would kill DataChannel-polyfill and work since now on native DataChannels because I got too much behind with them? Well...

...not so real :-)

Since the last post I have been thinking almost obsesively with the problem of needing a deprecated browser version just because of using my DataChannel-polyfill using WebSockets, and not working in latest versions with native support for DataChannels (that, by the way, sucks). So taking in account that the DataChannel-polyfill test code works with native ones I conclude that it would be just a problem on ShareIt! side not being fully compliant, so I went straight to it, removed all references to my polyfill and started again over with the handshake and PeerConnections layer from scratch, fixing one bug at a time.

And finally, after just some hours of work (instead of several days as I thought), it worked :-)

I must to admit that it's not the best implementation ever (I still ask my self why the h*ll is required to exchange some 'candidate' messages between the peers) and also it only support to transfer files and data up to 1Kb until they implement native reliable support (or I use reliable library, whatever comes first... :-) ), but the fact is that this almost mystical sensation after seen this thing working and with so few work (just some little tune-ups on the code thanks to the fact I develop DataChannel-polyfill trying to mimic the original specification API) it's really awesome :-D

So, not only I have experimental native DataChannels support on ShareIt! (F*ck yeah!!!!! :-D) but also being so little the diferences I've decided not only don't kill DataChannel-polyfill, but also try to improve it now that I know how native DataChannels work (and being so easy to use :-) ) and think about how to make a backward compatible API so older browsers could talk with newer ones :-)

But don't get me wrong little friends, the most important part of this post, if it was not clearly noticed yet, is the fact that now we don't need servers at all, being fully decentralized and serverless :-D Combine this with the Holy Grail of Internet that almost can bring us the HandshakeManager and the handshake servers, and wonder what an awesome future is waiting us... :-D

ShareIt! 1.1.1: Cloverfield

During the last weeks I was not able to publish too much here, specially because I'll start my exams the next week. But, since I got to be able to publish ShareIt! as my practice for my Networks-II class, I've been able to dedicate some time to it (maybe too much compared with the other asignatures... :-/ ) and develop some improvements directed by my tearcher Luis López, just in time not only for the lecture of the practice (tomorrow Monday on the morning :-P ) but also for the final phase of the championship, so here I'm showing to you the third public release of ShareIt! 1.1.1 "Cloverfield" (and no, although is the third release, the version is not written in unary ;-P)

It would be a fairly minor release because it only have two issues, but the fact is that it required some work on another ones and has added some great features and improvements on the architecture:

Manage duplicates: one of the problems that had ShareIt! previously is that since it was using the files hashes as unique identifiers of the files, if one of them was included twice only one of the copies whas showed (and served) also if it has different names. The solution was to identify them independently so both entries would coexist, and the ideal solution was to use the file's path. But what happens with the hashes? How I would be able to look for them if I'm not using them anymore as keys? I have been been putting off this thing a lot of time up to the point I didn't remember why I didn't do it before... and when I remembered it, I desired to forget it again.

IndexedDB: the reason why I have been been putting off so much the management of duplicates is because the optimal solution was to use IndexedDB's indexes, in fact a keyPath (multiple fields) for the primary key, and a non-unique index for the hashes (so they would be duplicated). What's the problem? Chrome hasn't still fixed the support of Blob and File objects inside IndexedDB (after almost one year and a half...), so I was alone using my polyfill. Two options: accept it, and told the reason to my teacher, or get mad myself, go for it and implement index support on my polyfill. At the worst, I would reuse later on my upcoming implementation of IndexedDB for Node.js for the headless version of ShareIt!... So I went directly for it, and after 8 hours listening to Daft Punk and the discography of Ke$ha and Demi Lovato (no comments), I got support for indexes on my own (in-browser, memory-only) pure Javascript IndexedDB implementation. #feelingLikeABoss... B-)

Sharedpoints: after that achievement another interesting point would have been detecting when a file has been added to a sharedpoint so it could be shared automatically without adding it by hand. Until this moment it was only able to detect when a file was deleted so it would be removed since I was using an input tag and its FileList object, that's mainly a photograph of the files that where available when the user selected it (although you can detect when it has beem removed checking an error when accessing to a file), and also it was done in a not maintable way, being all the management of the sharedpoints inside the hasher. Taking that in account, I decided to develop a new sharedpoint using DirectoryEntry from scratch, so I could be able to take in account the diferences and make it more portable. Now is fairly easier to add new types of sharedpoints like the ones I have been thinking about for Dropbox and Mega, and also I have been thinking about consider that internally the other peers would work as sharedpoints so I could simplify the way to access to them. Problem is, that I didn't notice until I finished it that Chrome has another bug not populating entries on the input tag and also I didn't be able to make drag&drop to work (this probably my fault...) so I've let this to be finished later.

And lastly by not least, the biggest feature added to this release and the one that made me think about tagging it as a mayor one, is the fact that now we have searches!!! :-D It's very primigenic and required some hacks (the fileslist is being shared between the peers after connection, having all of them the info of what are sharing others), but thanks to Lun-r.js now we can search by keywords, giving us a list of the files and hashes that correspond the them ready to be downloaded. It need some improvement in the future (like for example sending request to the others) but definitely it works :-)

But the reason why I didn't released it as a mayor release besides the fact that it has few (but important!!!) improvements, is that it will be the last of the 1.x series. This week I've discovered that ShareIt! doesn't work on Chrome browsers equal or newer than the current stable v26 because some API changes (the ones that almost crash the 1.1 release some weeks ago), so being only able to work on a legacy browser (Chrome v25) with my DataChannel-polyfill and without time in the next weeks to fix it because of the exams, I've decided to go directly to the 2.0 version after that and start using native DataChannels (I hope that by then reliable support will be implemented...), try to remove all the polyfills and develop all the backward incompatible features of the new protocol, like multiple and independent channels per PeerConnection on WebP2P or change the protocol to be RPC-based to increase reliability.

Good luck DataChannel-polyfill, and thank for the fish! :-D

WebP2P.io

I know that at this moment the WebP2P library is very tied to ShareIt!, specially because the universitary championship requires to have only one repository, but since I'll split it in the near future and in prevention of stealing it, I've decided to bought the (rather expensive...) domain WebP2P.io to host there the library project, focused in allow a server-less and distributed signaling channel for WebRTC applications. At this moment it only host a default webpage, but in the next weeks after I split the code I'll fill it with more info :-)

Yoigo no cuida sus clientes, verdad verdadera

Hoy voy a dejar de hablar de ShareIt! para hacer una queja publica, de ahi que la entrada sea en español.

Recientemente he tenido problemas con mi telefono movil, los cuales el operador Yoigo (con el que tengo contratada la linea) no ha querido hacerse responsable de ellos sino que encima me han tratado con prepotencia y desden y se han estado pasando responsabilidades unos a otros, motivo por el cual estoy muy seriamente decidido a cambiar de compañia (a no ser que ahora se me presenten con que me regalan un movil nuevo, lo cual haria pensarmelo segun cual sea el modelo...). Despues de un mes peleandome con ellos al final les critico en Twitter, momento en el que entran a razones y me piden que les envie un email contandoles mi problema. Pues bien, este es el email que les he mandado (cuidado, es largo):

Buenas tardes, tal y como os comente por Twitter, aqui muestro las
razones que me han llevado en menos de un mes de querer renovar mi
telefono movil con Yoigo e incluso recomendar Yoigo a mis conocidos
(especialmente porque es la unica compañia de las grandes con la que
no siento que me estafan con las tarifas), a estar decidido a cambiar
de compañia de forma inmediata.

Actualmente, poseo un telefono movil Samsung Galaxy SCL que me
ofrecisteis de regalo hace algo menos de año y medio como contraoferta
cuando intente cambiarme de compañia al terminar mi periodo de
permanencia. El motivo por aquel entonces fue que mi telefono habia
sido robado meses antes y desde atencion al cliente se desentendieron
del problema incluso despues de haber puesto una denuncia, por lo que
despues de varios meses pagando por un terminal y una conexion a
internet de los que no disfrutaba y tener que usar un movil antiguo
que tenia perdido por casa, decidi cambiarme para obtener un nuevo
terminal puesto que el internet en el movil se me habia hecho algo
imprescindible. Sin embargo, como contraoferta me ofrecisteis el
telefono antes citado totalmente gratis, y viendo que era un buen
detalle decidi continuar con vosotros.

Volvamos al presente. Dicho terminal empezo a darme problema de
bateria hara medio año, llegando dificilmente al final del dia. Es
normal, las baterias se ceban.. pero desde navidades ya empezo a darme
problemas mas serios: la interfaz se notaba pesada e incluso saltaba
el salvapantallas sin llegar a reaccionar, se sobrecalentaba en exceso
hasta el punto de no poder sugetarlo con la mano, se reiniciaba solo e
incluso a veces despues de tenerlo recien cargado, lo guardaba en el
bolsillo y al cabo de una hora notaba que me quemaba la pierna y la
bateria estaba completamente agotada. Sin embargo, aunque ya la
bateria me duraba unas 6 horas, desde semana santa la duracion ha
caido en picado hasta apenas 4 horas, ademas de que el terminal empezo
a hacer cosas raras como cortar llamadas (no me duran mas de dos o
tres minutos), pasar de estar con 4 rayas y 3G activado a un segundo
despues decir que no hay cobertura y un instante despues volver a
conectarse mediante roaming, bloquearse por completo teniendo que
sacar la bateria para reiniciarlo, apagarse y decir que la bateria
esta agotada pero despues indicar al recargar que esta por encima de
la mitad...

Debido a todo esto, empece a mirar terminales de Renuevo. Faltaban
menos de dos meses para el fin de la permanencia y estaba muy contento
con vosotros, asi que decidi continuar con vosotros. Mirando los
terminales que teniais disponibles vi que me ofreciais el Galaxy Note
II por 15€ al mes durante 24 meses, y puesto que no tengo tablet y me
vendria bien para estudiar los apuntes de la universidad, decidi ir a
por el.

Primer problema: mi telefono estropeado sigue en permanencia, y aunque
estuviera convencido de desear continuar con vosotros y ademas con una
permanencia de dos años y pagando un telefono movil caro, me indicais
que tengo que pagar la penalizacion de la permanencia del telefono
antiguo. ¡Pero si esta estropeado y no funciona, y ademas voy a seguir
con vosotros y no me voy a ir a la competencia! Idealmente al estar
permanencia y garantia lo suyo seria que se me ofreciera otro terminal
totalmente gratis en su lugar. Pues nada, que la unica solucion es o
pagar la penalizacion, o aguantar estos dos meses... Visto lo cual,
creo que lo mejor es intentar arreglarlo ya que no tengo ningun otro
terminal y ademas necesito estar conectado 24 horas por el trabajo y
las clases.

Segundo problema: en la tienda se me indica que la reparacion tardaria
25 dias, durante los cuales no podria disponer de mi telefono ni de
ningun otro ya que no tienen telefono de sustitucion disponible (¿no
pueden abrir una caja de los que tienen en la trastienda y dejarmelo?
Ademas, les serviria de demostracion y prueba de los nuevos terminales
y la gente podria convencerse de usarlos...), aparte de que me
dejarian uno antiguo que solo sirve para llamar con lo que estaria
pagando por un internet que no solo uso sino que necesito por trabajo.
Al no ser esta una solucion viable, decido llevarlo directamente al
servicio tecnico de Samsung, que no solo esta cerca de mi casa, sino
que ademas tardan solo una semana en lugar de los 25 dias que tardaria
dejandolo en tienda.

Tercer problema: el telefono lo recibi directamente por mensajero, sin
tener ningun documento que certifique que soy su propietario. Decido
acercarme a la tienda solicitando una copia de la factura la cual me
deniegan y me entregan un "Certificado de Activacion de IMEI"
indicandome que con eso puedo ir al servicio tecnico sin problemas. Es
lo que hago... y alli me indican que "ese papel no vale nada, no es
mas que una plantilla que rellenan con el Word en la tienda y le ponen
un sello, necesitan una factura aunque sea de cero euros al ser un
regalo o un albaran de entrega", y tambien que estan hartos y cansados
de tener que explicarles a los clientes de Yoigo siempre la misma
historia, dejando a los clientes insatisfechos por un problema que no
es suyo sino de Yoigo. Por otra parte, en el servicio tecnico me
indican cual es el problema y que la unica solucion es sustituirlo por
un terminal nuevo especialmente estando en garantia, ya que es un
fallo de diseño de fabrica de este modelo en concreto (la CPU y el
chip de memoria se calientan en exceso y se producen los fallos que he
indicado antes).

Cuarto problema: ante esta situacion, vuelvo a la tienda solicitando
una copia de la factura tal y como me indicaron en el servicio
tecnico, diciendome igualmente que son tonterias suyas y que deberian
aceptarme el certificado (la misma historia que el servicio tecnico
oficial de Samsung, pero cambiando los roles). Al ser incapaz de
ofrecerme una factura puesto que el terminal no fue vendido en tienda
sino por "internet" (realmente fue una contraoferta por telefono,
aunque supongo que se referiria a que internamente los procedimientos
serian los mismos en ambos casos...) que llamara a Atencion al Cliente
en la central (telefono 622). Una vez ahi, me confirman la situacion y
proceden a enviarme la factura, indicandoseme que la recibiria durante
la semana pasada. Al no tener noticias de vosotros decido volver a
llamar y me notificais que eso es totalmente imposible, puesto que
hace meses que no enviais facturas, que no teneis medios materiales
para hacerlo posible y que la chica que me atendio se debio confundir,
que si quiero una factura la solicite en tienda, aparte de decirme la
encargada de seccion que "tuviera cuidado" porque no sabia si al
utilizar un servicio tecnico distinto al de Yoigo lo cubriria la
garantia (¡pero si lo iba a llevar al servicio tecnico oficial!)
notandose en su tono de voz como una amenaza.

Quinto problema: otra vez en la tienda me cuentan la misma historia de
solicitarla a la central, momento en el que llamo directamente desde
la tienda para que se pongan de acuerdo y al final todo se aclara
resultando que no se dan facturas de ningun tipo, lo cual despues de
consultarlo en Consumo me notifican que es algo ilegal y que Yoigo
esta obligada a otorgarme una factura y permitirme llevar el terminal
directamente al servicio tecnico oficial si asi lo deseo. Sin embargo
tambien me notifica que el proceso tardara unos tres meses y lo mejor
que puedo hacer por el momento es aceptar el servicio tecnico de
Yoigo, el cual me dejara sin telefono durante casi un mes.

Sexto problema: visto lo cual, acepto mi derrota y ante la vision de
estar sin terminal decido pagar la penalizacion de la permanencia de
mi movil estropeado y solicitar el Galaxy Note II que queria... solo
para encontrarme que mi oferta especial ha desaparecido y que ahora el
terminal cuesta 20€ al mes, un 33% mas que antes, haciendo que tenga
un precio comparable a comprarlo libre y perdiendo la oferta de
cualquier interes. Quizas se pueda argumentar que sea por haber bajado
mi consumo ultimamente (lo cual me pareceria algo realmente rastrero),
pero lo cierto es que el hecho de "dosificar mis llamadas" ultimamente
es por el hecho de que no me duran mas de dos o tres minutos antes de
cortarse y apagarse el movil, motivo por el que solo hago las
imprescindibles. Asi pues, el ultimo punto que despues de toda esta
historia todavia me hacia plantearme el seguir con vosotros habia
desaparecido, asi que a partir de ahora ya no habia nada mas que me
mantuviera unico a Yoigo.

Asi que esta es mi situacion: un servicio de atencion al cliente que
me ningunea e ignora mi interes en querer seguir siendo cliente
vuestro, un servicio tecnico lento y deplorable que me da mas
problemas que soluciones, falta de coordinacion entre los distintos
miembros de la empresa "pasandose la pelota" unos a otros y delegando
responsabilidades en los demas y viendo recientemente que las "ofertas
exclusivas" no son tan exclusivas y que hay otras compañias ofreciendo
tarifas similares sin requerirme mantener permanencias de ningun tipo,
he tomado la decision de que cuando termine mi permanencia dentro de
poco mas de un mes (justo el tiempo que estara mi terminal en el
servicio tecnico) solicitare la liberacion de mi terminal y la
portabilidad a otra compañia mientras espero a que la reclamacion en
consumo me otorgue la factura para poder reparar y actualizar mi
terminal en los restantes 6 meses de garantia que me ofrece el
fabricante.

Sin otro particular y esperando su respuesta con sus impresiones al
respecto de mi sucesion de problemas con su compañia, se despide Jesús
Leganés Combarro.

A lo cual, solo se han dignado a responderme con un escueto, insulso y autogenerado mensaje:

Buenos días,

Por favor, facilítanos tus datos (nombre, apellidos, DNI y número de línea) para pasar tu correo al departamento correspondiente.

Gracias,
Un saludo.

En resumen, que siguen pasandose la pelota unos a otros, y ahora mi movil esta totalmente muerto durandole la bateria menos de 10 minutos desde esta mañana y estoy pagando por un linea y un servicio que no estoy recibiendo a pesar de estar el movil en permanencia y en garantia.

Definitivamente, Yoigo no cuida a sus clientes, verdad verdadera. Oh Amena, mon amour...

ShareIt! 1.1: Braveheart

Today is an epic day for ShareIt!. Lately I have been able to use the project for my networks class, and this "extra time" about working on in during my study time and also with the guidance of my teacher have lead me to make some great improvements on it, and now I'm proud to show you ShareIt! 1.1: Braveheart.

This new release not only has a lot of bugfixes, but also has a really better code structure that would allow to be easier to accept external collaborations and start adapting it to became a general purpose P2P framework, but also some great improvements on security, being the most important one the fact that now it uses by default anonimous XMPP servers to manage handshakes while being still compatible with the old system using PubNub, connect automatically to other peers on load to start creating a network mesh and speed up the connections, and not only that, now it's integrated a mobile version of the application, so if you access from your mobile phone or tablet you will be able also to use it and transfer files wherever you are. Also, all this things are ready not only for my tearcher's revision that I have in some hours that also for the keynote I'll have this afternoon at #JoinBiz10 :-D

So here you are, happy testing and don't forget that the user feedback, issues and pull requests are always welcome... :-D

ShareIt! working on anonimous XMPP (at last!!! :-D )

It was fairly more difficult to achieve that what I though at a first, but finally now it has been possible: ShareIt! is using an anonimous XMPP server to do the initial handshake instead of PubNub, so now it's not using a centralized server (it keeps the DataChannel-polyfill backend, but it's a matter of some more weeks until native support gets more stabilized) and it's using a distributed, federated network. I needed to change several things on the protocol and the implementation so now the inter-peers routing doesn't work, but I hope to fix it easily after changing WebP2P to be able to use several DataChannels at a time. This is really a HUGE step on the project viability, by the way, and just in time for the next milestone... :-D

ShareIt! on #JoinBiz10

 I have been selected to do the next monday a keynote (really, a 'lighting talk') on the #JoinBiz10, an entrepreneurship event where interesting IT projects can meet managers and MBAs and viceversa. Leets see if I can get some patrocinators for ShareIt!... :-D

And in other news, my networks teacher has accepted to use ShareIt! as final practice and has give me some guidance, and also I have just got to use XMPP instead of PubNub as handshake server. Please don't change the dial... :-P

P2P networks everywhere

A lot of people has said already that WebRTC and Peer-2-Peer comunications in the browser would be the next Internet revolution ("Internet 3.0", they call it), but I wouldn't suspect that this would start so soon: the next 7th April will be released PeerKit, the first WebRTC P2P enabled application (besides my P2P filesharing application :-P ), this case oriented as a distributed, zero cost Content Delivery Network. I don't know how effective or speed it would be (search a peer with the data, delegate the request...) and there's the problem of using the client connections to distribute the content (and with mobile phones connections this would be a serious problem), but don't doubt that if it works the server costs will drop abruptly world-wide... :-D

Slipping in the snowball

As you would know, I conceived originally ShareIt! as a real-time files-oriented PasteBin-like service using a WebSockets-based relay server (creating buzzwords the wild way :-P ). This was previously to known about WebRTC and DataChannels, that demostrated to be a way better aproach.

Obviously I wasn't the only want that saw this, and lately they are appearing a lot of other projects with a similar focus, like PeerJS, ShareFest (that recently got alpha status - congratulations! :-) ), OpenPeer or Drop&Load, and other related ones like DirtyShare, QuickShare or WhatAreYouDownloading. You could think about this that's a problem (too much related projects developed indepently and with their own incompatible protocols), and more taking in account that the WebRTC doesn't stop to being modified. What a mess! How something productive can be achieved with this perspective?

Obviously I wasn't the only want that saw this ;-) and that's the reason why in the latest weeks have been discussed to create a common P2P protocol that could make all this implementations compatible... and seems will be heavily influenced by ShareIt! 2.0 and WebP2P ideas :-D They agree on the point of split the handshake and the connection framework from the application specific protocols, so it could be easier to integrate with current protocols like Gnutella, and also it could be able to improve stability and versalite on the handshake servers (everybody is using custom ones, while it seems I'm just the only one that want to use generic third party channels... :-P).

Unluckily, I'm really busy lately with the classes up to the point I don't have time also for my work (I'll try to get up to date this spring week), so I believe I will not be able to do almost any progress on ShareIt! source code until summer, just reading others work on email. The good part is, by summer usable native DataChannels should be a reality... :-D

T3chFest Keynote

Finally, after two weeks while they were editing it, I've just received the video of the T3chFest presentation :-D


As you can see, the keynote was really well, without failures and a lot of question at the end (in fact I was almost the only one that got questions :-P ) although the limited time :-D Unluckily, I only got a third position. The reason? "It's not enought commercial, but you should think about develop it as a research project".

Taking in account that I was hearthly confident that I would get to start a company from ShareIt! and how is the situation of research on Spain, at a first point I toke the comment really bad, but on a second chance I though, isn't this was I always wanted? :-) To be honest, I've always wanted to work on R&D on my own projects, only that I need to pay the bills. That's the reason why I think that finally I'll do my career thesis with ShareIt! and work on it as an research project after that on my spare time (although it would be better to find a filantropist that believe in it! :-P ). In the last days I have been thinking about some interesting concepts for the next version of ShareIt! and WebP2P becoming them in a more general and ambitious project, and they'll need some attention and love to became true... :-)

if you can not beat them, get yourself a window and jumps

During the last six months I have been waiting for just-this-week. Why? Easy: last wednesday was the expiration day of the best domain for the project: share.it. The fact is that it was abandoned (although being a good one...) so I waited for it, but at the last moment it was renewed for one more year without any reason, and in fact currently is empty and also without a parking page. I tried to contact with it's current propietary without success, so the best option was to forget about it.

...until yesterday. I was reading the newspaper during the lunch time when I saw a full page ad from a new domain provider offering .es domains for 5€, and then got an idea: I checked out for share.it, shareit.com, shareit.net... without success, but NOT for shareit.es!!! So I quickly runned to the free-access room just to find that for inaugural promotion, the domains will be for a week at just 1€... and that the .es was free :-)

So here we are, I'm proud to show you the new ShareIt! home page, ShareIt.es:

It needs some clean-up and to fill it a little, but it a good start :-)

Also, if someone can reach me the infamous .it domain, I will thank him indefinitelly :-P

Admiration

With comments like this...

I have not the pleasure of meeting this guy personally, but I know his work and I can tell you that is available to very few. And when you consider their age and other circumstances, we can say that is one of a kind.
So what can I say. He is good and he knows it, his work is there and anyone who knows what this is all he knows. A wretched HR dauber come to waste your time and make stupid questions, I see it a lack of respect. The problem is from the HR department who have failed to assess their profile before interview. Does negotiate their contracts a top player like a regional division one? That's it.
And finally, I think you confuse arrogance with pride. If usually we would had a little bit of pride in this sector, sure we all would go much better.

...you, dear stranger, made me smile :-)

Ok, the Slashdot effect thanks to this publicity was not bad, too... :-P

Technical issues

Trying to add support for native DataChannels on Chromium v26 I have had some problems in the last days and ShareIt! was not working correctly, so after going back and fixing it I decided to do a Draconian decission: no more references to external libraries, and use instead a local copy. I know this would break one of the points of the Universitary Free Software Championship (re-use existent code), but the fact is that the external libs are all in an unique place (the /lib folder) so it can be easily upgraded, and also this has several advantages, like being sure to have a working version all the time since it's being used an expecific version of the libraries, being easier to deploy on any other place, and since all the code is auto-contained now I don't require to add some special permissions on the FirefoxOS manifest file, so users don't need to worry about them. This also has lead me to isolate the usage of the EventTarget.js library only to WebP2P, so now the UI uses exclusively jQuery events, making it more straigforward to develop and to use less memory usage. Now it only keeps to finish a script I'm working on to update all the external libraries at a time, so the fact of being the libraries inside the source code tree or as external libs would be indiferent.

Finalist on T3chFest!

With a punctuation of 19.2 over 20 and currently in the second position on the classification (the first one is for an universitary social network, that being the truth, I would love to see it being used as my university virtual campus... :-D ), I've been finalist on the T3chFest and the next friday I will present ShareIt! in front of a crowd of 70 persons and also compete for the final points for the first position :-D It will be recorded and also streamed, so I will upload the video when I get it :-)

A little bit more work, and the next step the final of the national Universitary Free Software Championship! :-D

When ShareIt! met Gnutella...

On the recent post on HackerNews has appeared a curious comment:

Hmm, it sounds like you're reimplementing Gnutella for the browser, which is not necessarily a bad thing!

I've never been a P2P guy since I was connecting to Internet using neigtbours wifi (and it was already dificult to do just plain e-mail and web surfing...), so the other times I read about how the diferent P2P protocols worked I only got the more superficial idea. That's the reason why this comment got my interest and take a look on wikipedia.

But this time it was diferent: I'm developing my own P2P protocol, so now I know what all that words means, and maybe just a trick of my unsconscious or maybe just serendipity, but I got a good surprise... both protocols and priorities ARE ALMOST THE SAME!!!! :-D This have two direct implications:

1. ShareIt! and the WebP2P protocol are not a total innovation "of my own", so this would be a little bit annoying (the "not invented here" complex... hey, what's the matter? I was raised using Macintosh... :-P )
2. on the other hand, having both projects the same concepts in mind I could learn from their errors, and also adapting the protocol to use theirs and creating proxies in the same way of the SIP-over-WebSockets ones, I would be able to access to the currently in active most used P2P network in the world by free :-D

I don't know, but by the moment, I'll try to send an email to the Gnutella guys to see what's happens... :-P

Unexpected consequences

Yesterday I published about WhatAreYouDownloading, isn't it? Ok: this morning his author ShirsenduK has finally uploaded his project to internet, but also it put a post on HackerNews about it. I can only say one thing:

What a madness!!! :-P

The fact is that in just some hours that's got crazy and people it's totally enthusiasmated about my project and they are forking, publishing and starring it on GitHub like there's no tomorrow!!! And the same it's happening with WhatAreYouDownloading. In fact, in just 8 hours I've got a 50% more stargazers than in the previous 6 months. Pfeu! :-P However, the most annoying thing I found reading the HackerNews comments (appart from some interesting links about networks and P2P security) was:

An in-browser BitTorrent client using my technology?!?! O_o

I don't know if I should get happy about building it, or get angry because nobody told me it... :-P

Also, one unexpected side effect of so much forks is that I didn't knew that they are done also with their branchs, so the production branch is also being copied... and automatically published by GitHub :-D This has lead me to the decision that it's the best for the project to split WebP2P on an independent project and improve the documentation to make sure that all this forks follow a common standard and can keep living, whatever finally happens with the championship regarding to the source code repositories.

Finally, the revolution it's starting, and I have first row tickets and a pop-corn bucket to see it... :-D

Dear Santa...

...although a little bit late (or very early, it depends how do you see it) I'm writing to you to say that I've been a good child and I want this t-shirt:

Or if not, I want a pony. Or both things. Whatever it's easier to you.

Loving, Piranna


:-P

The cake is NOT a lie

One month ago I published a post asking about if I could be able to get something better than a job thanks to ShareIt!... this week I got the answer to this question: that ShareIt! would be used as a basis for a product on a multi-national company :-D

This week I received an email from Shirsendu Karmakar, an engineer at SlideShare (recently bought by LinkedIn) telling me that he was impressed by ShareIt! and that he will use it for a free an open project. It's nice to know about it... but it's nicer that he give me administrator priviledges on the repository without asking about it :-P

So WhatAreYouDownloading it's mainly a fork of ShareIt! with a cleaner interface based on Bootstrap instead of plain jQuery (that by the way I was thinking about it at a first for UI homogeneity, althought it didn't fit too well with my original desing and I dropped it very early on the development process) but with a radical UI design more minimalistic (it reminds me to Windows 8 Metro) using dialogs instead of tabs for the diferent sections and, being sincere, with a better fit for mobile phones :-)

Main screen

Sharing "tab", now a dialog, the same as the other ones :-)

Also he linted the source code and did a code organization fairly interesting, using WebP2P as a library although I didn't finished yet to isolate it. What a nice piece of cake... :-)

The best of all, since it's just an aesthetic (and code) make-up, the WebP2P protocol is intact so WhatAreYouDownloading it's totally compatible with ShareIt! that was one of the design purposses I had in mind during development (allow others to develop their own applications being compatibles with other ones using the same protocol), so this made me think that I should split ShareIt! and WebP2P source code as soon as possible to allow more applications like WhatAreYouDownloading to appear :-) As a first approach, I'll try to re-merge the linted code inside ShareIt! source code tree and sync WhatAreYouDownloading interface to the latest modifications of ShareIt! code so they could interoperate better, let's see later where the rabbit's hole ends... :-P

Backing-up

One problems that I have found on the IndexedDB specification is the fact that databases has a same-origin policy that don't allow to share a database between several pages. So what happens if the domain of your page gets down? If it's a normal webapp with a central database and IndexedDB is used as a cache not much, just that you'll need to re-sync your content. But what happens with a server-less, pure client-side webapp like ShareIt!? There's no central database where to sync, so YOUR data is isolated and inacessible in YOUR computer, and also according to the IndexedDB specification it can be deleted by the browser at any moment. Not a good thing.

A solution that was given to me was to do some tricks with the /etc/hosts file and load a local copy of the webapp, or at least a dumper application. Nice and intelligent trick, but not available for non-techies users. What's the other (I hope that temporal) solution? Implement a backup system for the cache. This way, the cache content can be stored on a zip file directly from inside ShareIt! and later used to restore it on the same computer or on your new bough machine, or also moved to another one and update its cache with the content of your backup and help your friends to finish their downloads :-)

Now I must to go, it's time to see what Kim DotCom has to show us... :-D


P.D.: I need to see if with AppCache being the application data loaded from localhost this happens too, but I'm not sure...

Handshake over transport channels

Just some minutes ago got to be enough mature one of the piece of cake that I've been working on the last two weeks: clean-up of the handshake signaling code and adapted the handshake servers interface to be more like Message Channels. This way it can be used with regular channels like the ones from PeerConnection DataChannels so this way a peer can work like a regular handshake server enabling a signaling channel to connect two peers thorough it, and as a collateral damage, peers can be found over all the network mesh without requiring a handshake server on the cloud at all.

Wait, has you developed a distributed discovering routing protocol for browsers?!?!

Yes :-)

Since using a cloud service like PubNub is not the best idea regarding to the future of the network being the biggest point-of-failure of all the platform, the best idea is to move the peers routing to the webp2p network as soon as possible. To do this, when a peer get connected to one of the handshake server it keeps waiting until some more have been connected after it, giving them an offer to be connected to the network through them just like "reverse-proxies" (in the same way they were connected before), and when they get enough peers connected through them, they get disconnected. This way we can be sure that the network is sufficiently dense but also increase the mesh entropy not being all the peers connected through the same peers, making it more difficult to break down.

Also, since I have changed the signaling code to be a transport interface now the handshake servers code are as minimal as possible so it's easy to interchange with another one. Being the signaling (now routing) code a transport interface it can be used over DataChannels, so the next step was to know how to find the peers on the mesh. The easiest way it's just with a flooding protocol. It's not the most optimal regarding network resources, but since KadOH it's not currently ready (it mainly seems it's dead... :-/ ) at least it's easy to develop and a good starting point to start investigating from here, and it can find a peer over all the network if it exist (I need to think what to do if it doesn't exists... maybe using a timeout?) that's the interesting point. Now to know how a peer can get back connected I've done the easiest way to achieve it: just to append a list of all the peers where the "packet" have been routed, that can also be used to stop the propagation of the flood, so I get two birds with just one shoot :-D

Now the single-point-of-failure that's the handshake server still exists there (I hate you >:-( ), but at least  it will be pass so little time that a peer will be connected there before routing only over the webp2p network that trying to catch one of them will be almost be an homenaje to Christina Rosenvinge... :-P

How to hack a contest

During the last two weeks until the end of the year there was a contest on the Youth Card of Madrid webpage about finding their mascots dressed as the Three Wise Men.

The fact is that I have been looking for them and only found two of them, so I decided to make a mirror of the page and looking for locally... just to find that in fact there were only two of them ¬¬ After sending them an email to know if it was a bug or a bad trick, they confirmed me that the third one only will appear from time to time since she (it's the girl with glasses ;-) ) is very shy, but will appear at least one time until the end of the year on the morning. I didn't wanted to be pushing F5 all the time but I want to win one of the prices, so I decided to do it my way.

The first of all is to be able to know how to identify if one of the mascots is on the page, so when I founded one of them, quickly I went to the Chrome Inspector to look on the page content, seeing references to a nice "navidad2012" folder and that the animated icons where made on Flash. Now I know what to look for, so now it's time to know where to look for. Since the page is mainly static (bizarre, I know, but what would you expect from a gubernamental institute? :-D ), for this I've made a mirror of the website on my local harddisk using wget so I could be able to look on their files for that string. Since wget show a lot of output, I'll silence it with the quiet argument:

wget -mq http://$BASE_URL

and later, searched on all the files using grep looking for the previous checked folder and for the Flash files, getting only the path of the files that have them:

grep -lR 'navidad2012/.*\.swf' $BASE_URL/*

Now it's time to get notified by mail when the pages where found. This was a little bit complicated since GMail protect itself for sending spam, so some additional config steps are required so we send the emails authentificated with our own account. First of all, we need to install the mailutils package for the mail command and ssmtp package for the output mail server:

sudo apt-get install mailutils ssmtp

and later configured the smtp server following these instructions (I didn't need to follow the steps regarding removing sendmail, so maybe it's not yet necesary at all). After that, we now are able to send emails from the command line (content is introducced via the standard input):

mail -s "$BASE_URL" "$EMAIL"

But receiving an email each time it checks it is not a good idea since we are only interested to check it when we have the Three Wise Men, so we'll check it before:

FILES=$(grep -lR 'navidad2012/.*\.swf' $BASE_URL/*)
WISE_MEN=$(echo "$FILES" | wc -l)

if [ $WISE_MEN -eq 3 ]
then
   ...
if

Now it's time to let this run each 5 minutes. A simple infinite loop and sleep would be enough, but removing the useless files if we didn't found what we wanted would be nice (after being sure to don't remove them using break on the condition)...:

while :
do
   ...
   rm -r $BASE_URL
   sleep 5m
done

Et voilá! Now it only needs to add some echo messages to now how it's working when I'm on the machine (mainly studying for the exams I have after holydays... ¬¬), remove all data before starting to work, a shebang and execution permissions and now I can be able to go to the gym without worrying to don't get the prize :-D

Full code of the script:

#!/bin/bash

BASE_URL=www.carnejovenmadrid.com
EMAIL=piranna@gmail.com

rm -rf $BASE_URL

while :
do
    wget -mq http://$BASE_URL
    FILES=$(grep -lR 'navidad2012/.*\.swf' $BASE_URL/*)
    WISE_MEN=$(echo "$FILES" | wc -l)

    if [ $WISE_MEN -eq 3 ]
    then
        echo
        echo "*** FOUNDED 3 WISE MEN!!! ***"
        echo "$FILES"

        echo "$FILES" | mail -s "$BASE_URL" "$EMAIL"
        break
    fi

    echo "Only found" $WISE_MEN "wise men at"
    echo "$FILES"
    echo "Removing files"
    echo

    rm -r $BASE_URL
    sleep 5m
done