![Joi Ito [logo]](/_site/img/joi-ito-logo-300.png){kind=logo}
I've been getting a lot of attention for my moblog even though it's not such a big technical feat really. All you need is a script to convert email into MT entires. We're going to make the code available so that hopefully more people can experiment with moblogging. It's being made available under a standard GPL v.2 license. Please let me know what you think. We hope to continue to improve it. The file is available at: http://joi.ito.com/archives/src/mail2entry-2002-12-25.tgz
Here are some notes about installation.
WarningThanks for all the work on this SenThis is a first attempt -- there are some things missing, but someone w/appropriate UNIX experience should be able to succeed in installation and configuration. I may update the instructions as I get feedback.
Prerequisites
The ability to read English and follow simple instructions.
A host connected to the Internet which can receive email [1] and:
Python >= 2.2.2
MT
enabled w/ the XML-RPC interface
a blog (determine your blog id)
a blog user, the associated password [2]Please note that this code was designed to run on the host which the blog itself is running on. Other configurations may be possible, but I don't recommend them and don't plan on supporting them at the moment.
Please send feedback to me for the moment. We may designate someone to be in charge of the code later. Also, since it uses XML-RPC, it should be easy to get it to work with other blog platforms.
PS The GPL license on this code overrides the CC license on this site.
Of course Radio and Blogger both have mail to post features...
btw, I have written similiar tool for Userland's Manila, see sample working at nipernaadi.kolhoos.ee with 690 pictures from around Estonia, taken by radio reporters this summer. Although the site is in Estonian language I wrote English description for the tool and I think there are couple of things worth stealing: for example automagically created icons, macro to display 10 random icons (with links to original images), option to "flip" pages where images + messages are stored etc.
Some problems right now with the code. The manual pipe doesn't work now.
aissata# cat /home/karl/essai | python mail2entry.py > toto
mail2entry.py:8: SyntaxWarning: import * only allowed at module level
def main():
Traceback (most recent call last):
File "mail2entry.py", line 50, in ?
main()
File "mail2entry.py", line 43, in main
return result
UnboundLocalError: local variable 'result' referenced before assignment
toto has the right format with a jpeg file in it.
Content-Type: image/jpeg
Content-Disposition: attachment; filename="essai.jpg"
Content-Transfer-Encoding: base64
Any ideas of the problem
First problem has been solved. But it seems that the program react very differently depending on the mailer. I had for example a message about UTF-8.
My problems seems now to be related to xmlrpc.cgi
Mobicarnet - first french Moblog
The pictures are saved as now.jpg and the secondary name but the post doesn't work yet. I'm trying to work out why.
Traceback (most recent call last):
File "/my_full_path/web/mobicarnet/mail2entry.py", line 34, in main
result = postentry.post(content)
File "postentry.py", line 38, in post
content, publish)
File "/usr/local/lib/python2.2/xmlrpclib.py", line 821, in __call__
return self.__send(self.__name, args)
File "/usr/local/lib/python2.2/xmlrpclib.py", line 975, in __request
verbose=self.__verbose
File "/usr/local/lib/python2.2/xmlrpclib.py", line 840, in request
self.send_content(h, request_body)
File "/usr/local/lib/python2.2/xmlrpclib.py", line 876, in send_content
connection.endheaders()
File "/usr/local/lib/python2.2/httplib.py", line 695, in endheaders
self._send_output()
File "/usr/local/lib/python2.2/httplib.py", line 581, in _send_output
self.send(msg)
File "/usr/local/lib/python2.2/httplib.py", line 548, in send
self.connect()
File "/usr/local/lib/python2.2/httplib.py", line 941, in connect
sock.connect((self.host, self.port))
error: (61, 'Connection refused')
Looking at the error you posted, I guess that some network connection is failing to establish -- note:
error: (61, 'Connection refused')
which is the last line of your posting.
Some number of lines above, the xmlrpc library is referenced, so one might guess that the problem lies in trying to establish a connection w/ the XMLRPC portion of MT.
If this guess is correct, there could be a number of different explanations for the current situation. Perhaps it would be worth testing whether XMLRPC is working for your MT configuration [1].
HTH
[1] It looks like XMLRPC capability is supposed to be enabled for this program to work -- at least the entry text lists:
MT
enabled w/ the XML-RPC interface
as one of the requirements.
it is working. I will complete your INSTALL manual to explain a few things
Thanks
* Some testing and examples
- Email that doesn't work at all.
in the mail headers
Mime-Version: 1.0
Content-Type: multipart/mixed;boundary="==IMail_v5.0=="
And after in the body
--==IMail_v5.0==
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Here Text
--==IMail_v5.0==
Content-Type: application/octet-stream; name="chalet.jpg"
Content-Transfer-Encoding: base64
here the coded information
For this email, the image is not saved and the post doesn't happen at all.
the tracelog says
Traceback (most recent call last):
File "mail2entry.py", line 28, in main
content, imagefilename = parsemsg.parse(sys.stdin)
File "parsemsg.py", line 65, in parse
imagefilename = handle_image_portion(second_part)
File "parsemsg.py", line 79, in handle_image_portion
raise SecondMIMEPartNotImage
SecondMIMEPartNotImage
* BodyNotUTF8 Problems
With mail with this header
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="------------070602060307040204040407"
and with this body
This is a multi-part message in MIME format.
--------------070602060307040204040407
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Here Text
--------------070602060307040204040407
Content-Type: image/jpeg;
name="tristan-mobicarnet.jpg"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="tristan-mobicarnet.jpg"
Here the encoding
The image is saved but the text is not posted to the blog.
the tracelog message is
Traceback (most recent call last):
File "mail2entry.py", line 34, in main
result = postentry.post(content)
File "postentry.py", line 25, in post
raise BodyNotUTF8
BodyNotUTF8
A few other things for the people who install it.
settings.py
"""Profile settings"""
username = "Photo User"
password = "somethingrelevant"
blogid = 3
publish = True
uri = "https://blog.example.org/cgi-bin/mt-xmlrpc.cgi"
"""Be careful, some servers do not have https authorized, you can use http simply instead"""
imagesdirpath = "/var/www/blog/archives/images"
imageurldir = "/blog/archives/images"
"""Be careful to not have a too long name. Images have already a long name and if the directory name is too long, some browsers will not be able to load the URI"""
template = "
"
"""Change the template mark-up for span instead of div. Right now if you leave it like that the page will generate an invalid code by putting p around div."""
logfilepath = "/tmp/log-tracebacks.log"
* Mail redirection in sendmail.
Some sendmail installation does not authorize the piping at the /etc/mail/aliases level or /etc/mail/virtusertable. In this case, a possible solution is to put the piping in .procmailrc in the directory of the email address if it's a user.
like
:0 Hc
*
|/usr/local/bin/python /Path/mail2entry.py /yourMoblogPath/img
Hope it helps.
OK I think I start to understand the email parsing problems. I have read the source of your parsemsg.py. You made it too specific for Japanese community. You should make it more international.
I can read for example
def parse(fileObject):
"""Parse a MIME message w/ two parts, where the first one contains
text (ISO-2022-JP or ASCII) and the second one contains an image,
w/ an optional subject header (decoded to ISO-2022-JP or ASCII).
Successful parsing should return a tuple w/ a content dictionary
suitable for posting and filename for a file containing the extracted
image."""
Many emails in the world are now posted in iso-8859-1 or UTF-8
So I think one of the first think to fix is to handle more common encoding:
def unicodify(string, charset):
"""Converts an ISO-2022-JP or US-ASCII string to unicode. A string using
another charset results in an empty string being returned."""
result = ""
if charset:
charset = charset.lower()
if charset == 'iso-2022-jp':
result = unicode(string, "japanese.iso-2022-jp")
elif charset == 'us-ascii':
result = unicode(string, "us-ascii")
else:
result = ""
else:
result = unicode(string, "us-ascii")
return result
For testing purpose it would be good to have a flag to just have an output of the two part (1 file for the image and 1 file for the text output) and a verbose mode of error on the console.
I'm travelling now, but I'm back home on Monday, I will be able to start working on the code to make it more tolerant and general.
Does it suit your needs?
Thanks Karl!
If anyone wants something similar that uses perl instead of python (and doesn't mind getting their hands just a little dirty editing the perl to taste), I've got one here. Just get rid of any lines with decode_iso_2002_jp in them, create a suitable .blogpostrc file and add a recipe to your procmail.
See also system overview. It's a crock but it works :)
Thanks for this Dav. I put it in the Moblog Resources Page
I've been trying to set this up on my server but I'm a new hand at installing scripts. I saved a message in the correct format to use as a test file but when I run the test I'm getting an error. I can't determine what the problem is myself...hopefully someone here will be able to nail it down for me.
bash-2.05$ cat testmoblog.eml | python /home/postpunk/mail2entry/mail2entry.py /home/postpunk/mail2entry/settings/
Traceback (innermost last):
File "/home/postpunk/mail2entry/mail2entry.py", line 50, in ?
main()
File "/home/postpunk/mail2entry/mail2entry.py", line 37, in main
lfo.write(time.strftime("%Y-%m-%d %H:%M\n\n"))
TypeError: function requires exactly 2 arguments; 1 given
The error "TypeError: function requires exactly 2 arguments; 1 given" indicates that you're running an older version of Python. Joi's script requires >= 2.2.2, mainly for the e-mail handling functions. I've learned this the hard way. :( Anyone using a good, cheap hosting company running a respectable version of Python?
i'm curious, does your mail2entry script constantly check for mail? check at a certain interval? or does it automatically know when the mail is recieved?
Successful parsing should return a tuple w/ a content dictionary
suitable for posting and filename for a file containing the extracted
image."""
Hi, I have a big problem with mail2entry. I'm new to python and have no idea of the problem happening... Here my mail.log:
Dec 11 02:55:55 tribu qmail: 1071107755.875636 new msg 7506064
Dec 11 02:55:55 tribu qmail: 1071107755.875768 info msg 7506064: bytes 3644 from qp 23522 uid 64011
Dec 11 02:55:55 tribu qmail: 1071107755.896792 starting delivery 966: msg 7506064 to local ga-fl.net-cl@ga-fl.net
Dec 11 02:55:55 tribu qmail: 1071107755.897087 status: local 1/10 remote 0/20
Dec 11 02:55:55 tribu qmail: 1071107755.964642 delivery 966: deferral: /usr/lib/cgi-bin/blog/moblog/mail2entry.py:8:_SyntaxWarning:_import_*_only_allowed_at_module_level/__def_main():/Traceback_(most_recent_call_last):/__File_"/usr/lib/cgi-bin/blog/moblog/mail2entry.py",_line_50,_in_?/____main()/__File_"/usr/lib/cgi-bin/blog/moblog/mail2entry.py",_line_12,_in_main/____settings_path_=_sys.argv[1]/IndexError:_list_index_out_of_range/
11 02:55:55 tribu qmail: 1071107755.964642 delivery 966: deferral: /usr/lib/cgi-bin/blog/moblog/mail2entry.py:8:_SyntaxWarning:_import_*_only_allowed_at_module_level/__def_main():/Traceback_(most_recent_call_last):/__File_"/usr/lib/cgi-bin/blog/moblog/mail2entry.py",_line_50,_in_?/____main()/__File_"/usr/lib/cgi-bin/blog/moblog/mail2entry.py",_line_12,_in_main/____settings_path_=_sys.argv[1]/IndexError:_list_index_out_of_range/
Dec 11 02:55:55 tribu qmail: 1071107755.964762 status: local 0/10 remote 0/20
Have you any idea for "list index out of range"?
Thanks
1. SyntaxWarning:_import_*_only_allowed_at_module_level
Line 17 that says "from settings import *" should be changed to name all the fields that need to be imported.
As such:
"from settings import username, password, blogid, publish, uri, imagesdirpath, imageurldir, template, logfilepath"
2. UnboundLocalError: local variable 'result' referenced before assignment
If an error occurs before line 34, the variable result has not been assigned yet and thus the eror. I inserted a "result = 1" on line 11 which fixes the error.
3. I had to modify parsemsg.py to accept iso-8859 or else it was causing 'BodyNotUTF8' errors:
On line 126 I inserted
elif charset == 'iso-8859-1':
result = unicode(string, "iso-8859-1")