Demand for end-to-end secure messaging has been growing rapidly and companies have responded by releasing applications that implement end-to-end secure messaging protocols. Signal and protocols based on Signal dominate the secure messaging applications. In this work we analyze conversational security properties provided by the Signal Android application against a variety of real world adversaries. We identify vulnerabilities that allow the Signal server to learn the contents of attachments, undetectably re-order and drop messages, and add and drop participants from group conversations. We then perform proof-of-concept attacks against the application to demonstrate the practicality of these vulnerabilities, and suggest mitigations that can detect our attacks. The main conclusion of our work is that we need to consider more than confidentiality and integrity of messages when designing future protocols. We also stress that protocols must protect against compromised servers and at a minimum implement a trust but verify model.